Deploying an Ephemeral Parallel File System for High Performance Computing
Building public cloud High Performance Computing (HPC) environments that require fast parallel file system access hasn’t always been easy. While it’s a relatively simple task to spin up hundreds or even thousands of instances in AWS, if they all have to share the same data - as is generally the case in HPC - standard file serving solutions such as AWS Elastic File System (EFS) don’t typically scale to meet the requirement of keeping the compute running at peak utilization.
Performing HPC in public clouds hasn’t always been cost-effective versus on-premise solutions since its workloads are so CPU intensive. Large research institutions, for example, do a very good job of keeping their own HPC infrastructure busy around the clock. But if the workloads are ephemeral - that is they come and go for running occasional jobs or for managing peak loads - they should spin up or down as quickly and efficiently as possible to minimize cost. In such cases bursting HPC workloads into public cloud HPC can be very compelling.
With growing requirements for on-demand HPC computing driven by rapid advances in industries such as biotech and technologies like machine learning, the need for a cost-effective ephemeral high- performance file system that can be shared among large numbers of compute instances has become increasingly obvious for use in the cloud.
Launched in November 2018, Amazon FSx for Lustre provides a high-performance file system designed for fast processing of workloads. This white paper aims to provide the reader with a framework for building a simple pipeline for the creation of an ephemeral FSx for Lustre file system to be used for compute-intensive environments.
“It has been my experience that building HPC environments - all of which require fast and parallel file system access - has never been a simple or easy task. Our work with FSx for Lustre finally enables, and more importantly automates, the timely cost-efficient creation and tear down of HPC in the Cloud.”
- Ludovic Francois, CEO, TrackIt
Applicability across industries
2- Media & Entertainment
Animation & VFX Rendering
Processing & Transcoding
3- Product Design
Electronic Design Automation
4- Classic Supercomputing
Molecular & Particle Simulation
5- Emerging Use Cases
Internet of Things Analytics
Lifecycle Pipeline for FSx For Lustre
FSx for Lustre requires the source data files to be on S3. If your files are not already there, common data placement options include:
AWS S3 Transfer Acceleration
AWS Data Sync
AWS Storage Gateway
Files are subsequently pushed to a dedicated FSx for Lustre bucket, with an S3 prefix for the current project.
FSx for Lustre Creation
The pipeline for the creation of the FSx for Lustre file system is illustrated in the AWS Design section below.
Notification of Creation to the User’s API
During the creation pipeline, different API calls can be made to monitor the creation of the file system. In the pipeline illustrated below, there are API 1 and API 2. Both APIs will be notified once the creation of FSx for Lustre is initiated and then again once the creation of the file system is complete.
Discover the full TrackIt white paper & learn more about FSx for Lustre by downloading this PDF file.
TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Marina del Rey, CA.
TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.
TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build and deploy a customized solution tailored to your exact requirements.
In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.