Categories

 Automation AWS data rooms Data Storage DevOps DevTools Monitoring Optimization Services TrackIt

Tags
ai ami API Aurora autoscaling aws AWS EC2 AWS S3 chargify Ci Cd Pipeline Circleci cloud Cloud Computing Cognito consul cost optimization dedicated DevOps docker Dynamodb ec2 ESXi github hashicorp high availability IP json Marketing nomad open source packer pfSense Postgres public cloud reduce cost Serverless terraform trackit tutorial Video VM vmware windows Windows Containers

Nov. 25, 2020

—

Data Pipeline Implementation

One of TrackIt’s clients had implemented a big data pipeline running on AWS that needed to be optimized. The client was leveraging the big data pipeline to enable its data scientists to gain additional insights by exploiting data that originated from CSV files.

This article details the TrackIt team’s approach to optimizing the architecture of the data pipeline.

Data Pipeline Implementation