Ingestion
FSx for Lustre requires the source data files to be on S3. If your files are not already there, common data placement options include:
- SFTP
- AWS S3 Transfer Acceleration
- AWS Data Sync
- AWS Storage Gateway
Files are subsequently pushed to a dedicated FSx for Lustre bucket, with an S3 prefix for the current project.
FSx for Lustre Creation
The pipeline for the creation of the FSx for Lustre file system is illustrated in the AWS Design section below.
Notification of Creation to the User’s API
During the creation pipeline, different API calls can be made to monitor the creation of the file system. In the pipeline illustrated below, there are API 1 and API 2. Both APIs will be notified once the creation of FSx for Lustre is initiated and then again once the creation of the file system is complete.
(...)
Discover the full TrackIt white paper & learn more about FSx for Lustre by downloading this PDF file.
About TrackIt
TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Marina del Rey, CA.
TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.
TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build and deploy a customized solution tailored to your exact requirements.
In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.
For more information, readers can visit www.TrackIt.io.
