TrackIt – Cloud Consulting & S/W Development
  • Home
  • Solutions
    • Media & EntertainmentMedia & Entertainment
    • HPC – High Performance ComputingHigh Performance Computing
    • Software DevSoftware Development; Serverless, CI/CD, UI/UX, API Integration
    • Migrate to AWS
    • DevOpsDevOps, AWS Cloud Consulting
    • StorageData Storage
    • Financial Services
  • Services
    • AWS Services
      • AWS Well-Architected Framework
      • Focus on Video AI/ML
      • Chime SDK
      • DevOps Guru
      • Virtual Desktop/Studio in the Cloud
      • Amazon API Gateway
      • AWS Lambda
      • Amazon DynamoDB
      • Amazon EKS
      • Amazon OpenSearch
      • Amazon CloudFront
      • AWS Elemental MediaConvert
    • Hammerspace Integration
    • Focus on Penetration Testing
    • Autodesk Service Provider
  • Resources
    • Blogs
    • White Papers
    • Case Studies
    • Press Releases
    • Trainings and Workshops – Unlock TrackIt’s Power
  • Tools & Apps
    • TrackIt Cost Management App
    • Tagbot
    • AI Video Reviewer
    • Trackflix
  • Managed Services
  • About Us
header image

TrackIt Blogs

Read the latest cloud technology news and learn new tips and tricks to get the most out of your cloud.

  • Contact Us

Categories

Automation AWS data rooms Data Storage DevOps DevTools Monitoring Optimization Services TrackIt

Tags

ami API Aurora aws AWS EC2 AWS S3 AWS Studio in the Cloud chargify Ci Cd Pipeline Circleci cloud Cloud Computing Cognito consul cost optimization dedicated Demonstration DevOps docker Dynamodb ec2 ESXi github Hammerspace hashicorp high availability HP Anyware HPA Tech Retreat 2023 HPTeradici json Marketing Media Workflows nomad open source packer pfSense Postgres public cloud reduce cost Remote Serverless terraform tutorial Video Windows Containers

Jun. 03, 2020

—

White Paper – Focus on FSx For Lustre



  • Version
  • Download 29
  • File Size 765.29 KB
  • File Count 1
  • Create Date June 3, 2020
  • Last Updated January 9, 2023
Download
Description

AWS FSx for Lustre

Here you can download the AWS FSx for Lustre white paper in a PDF file.

Deploying an Ephemeral Parallel File System for High Performance Computing

Building public cloud High Performance Computing (HPC) environments that require fast parallel file system access hasn’t always been easy. While it’s a relatively simple task to spin up hundreds or even thousands of instances in AWS, if they all have to share the same data - as is generally the case in HPC - standard file serving solutions such as AWS Elastic File System (EFS) don’t typically scale to meet the requirement of keeping the compute running at peak utilization.

Performing HPC in public clouds hasn’t always been cost-effective versus on-premise solutions since its workloads are so CPU intensive. Large research institutions, for example, do a very good job of keeping their own HPC infrastructure busy around the clock. But if the workloads are ephemeral - that is they come and go for running occasional jobs or for managing peak loads - they should spin up or down as quickly and efficiently as possible to minimize cost. In such cases bursting HPC workloads into public cloud HPC can be very compelling.

With growing requirements for on-demand HPC computing driven by rapid advances in industries such as biotech and technologies like machine learning, the need for a cost-effective ephemeral high- performance file system that can be shared among large numbers of compute instances has become increasingly obvious for use in the cloud.

Launched in November 2018, Amazon FSx for Lustre provides a high-performance file system designed for fast processing of workloads. This white paper aims to provide the reader with a framework for building a simple pipeline for the creation of an ephemeral FSx for Lustre file system to be used for compute-intensive environments.

“It has been my experience that building HPC environments - all of which require fast and parallel file system access - has never been a simple or easy task. Our work with FSx for Lustre finally enables, and more importantly automates, the timely cost-efficient creation and tear down of HPC in the Cloud.”

- Ludovic Francois, CEO, TrackIt

Applicability across industries

1- Biotech

  • Genomic Sequencing
  • Molecular Microscopy

2- Media & Entertainment

  • Animation & VFX Rendering
  • Processing & Transcoding

3- Product Design

  • Electronic Design Automation
  • Mechanical Simulation

4- Classic Supercomputing

  • Fluid Dynamics
  • Molecular & Particle Simulation
  • Weather

5- Emerging Use Cases

  • Machine Learning
  • Artificial Intelligence
  • Internet of Things Analytics

 

Lifecycle Pipeline for FSx For Lustre

Ingestion

FSx for Lustre requires the source data files to be on S3. If your files are not already there, common data placement options include:

  • SFTP
  • AWS S3 Transfer Acceleration
  • AWS Data Sync
  • AWS Storage Gateway

    Files are subsequently pushed to a dedicated FSx for Lustre bucket, with an S3 prefix for the current project.

    FSx for Lustre Creation

    The pipeline for the creation of the FSx for Lustre file system is illustrated in the AWS Design section below.

    Notification of Creation to the User’s API

    During the creation pipeline, different API calls can be made to monitor the creation of the file system. In the pipeline illustrated below, there are API 1 and API 2. Both APIs will be notified once the creation of FSx for Lustre is initiated and then again once the creation of the file system is complete.

    (...)

Discover the full TrackIt white paper & learn more about FSx for Lustre by downloading this PDF file. 

 

About TrackIt

TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Marina del Rey, CA.

TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.

TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build and deploy a customized solution tailored to your exact requirements.

In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.

For more information, readers can visit www.TrackIt.io.

FSx for lustre

 

 

 

Previous Next

AWS Advanced Consulting Partner - Cloud Management, Consulting, and Software Development Solutions

Quick links

    • Home
    • Solutions
    • Services
    • White Papers
    • Blog
    • About Us

Tools & Apps

    • TrackIt AI Video Reviewer
    • Trackflix
    • TrackIt App
    • Tagbot

get in touch

    • Phone: +1 323-230-0042
    • Email: info@trackit.io

    • TrackIt 578 Washington Blvd #260 Marina Del Rey, CA 90292
  • @TrackIt 2022