How to Build a VOD Pipeline on AWS

The Growing Need For A Simple VOD Solution

At TrackIt, we have experienced a growing number of content creators looking for an easy to use and simple VOD solution they can implement to start sharing their content with users.

As of September 2019, there was no offering currently available in the AWS marketplace that serves as a full-package plug-and-play VOD solution. Recognizing the need, TrackIt has developed and shared a clean and simple VOD workflow using AWS Elemental that can be quickly deployed with minimal technical expertise.

Challenges For Content Creators

1. Limited Control Over Video Content

When using AWS, content creators typically choose to store their video  content in S3 buckets. Unfortunately this provides very little control over the content related to the video. For example, a content creator can easily add tags but cannot add more (video metadata) information about the video such as location, name of the video creator, etc.

2. Potential Security Risks

In order to allow viewers to access the videos from a website, content creators
will often implement one of two scenarios:
1. Make the S3 bucket public: In this case everyone can access all of the
content – including private videos.
2. Add specific policies to the bucket: This approach is prone to mistakes in
configuration that expose the content creators to possible security
breaches.

As a result, using these approaches for a website to provide viewers with access to video content risks opening backdoor access into their AWS accounts

3. Issues With Video Quality/Resolution

In the absence of a purpose-built VOD solution, content creators generally only provide their viewers with videos of a single quality – the original quality of the uploaded video. Thus viewers with low bandwidth connections might have  trouble buffering the content, and those with high bandwidth don’t have the  option of experiencing higher-quality video.

The AWS Elemental VOD Workflow

TrackIt’s workflow has been designed to be an easy to implement and elegant  VOD solution. Our workflow implements only the minimum services required.  An API gateway is used to make requests, S3 buckets are used to store video content, and AWS Elemental MediaConvert transcodes content and manages  format resolutions.

AWS Elemental MediaConvert

AWS Elemental MediaConvert is a file-based video transcoding service that allows users to easily create video-on-demand (VOD) content for broadcast and multiscreen delivery at scale. Using MediaConvert allows content creators to provide users with multiple resolutions for each of the videos. 

How It Works

AWS Elemental MediaConvert is used for video transcoding. The source video is converted into two different kinds of output: MP4 and HLS (allowing users access to multiple resolutions they can switch between during playback). AWS Elemental MediaConvert also generates thumbnails by extracting multiple evenly distributed frames in the video to provide a preview capability.

Amazon CloudFront

Amazon CloudFront is AWS’ content delivery network (CDN). CloudFront provides content creators a simple and straightforward solution for video delivery with in-built caching mechanisms. A content creator simply needs to create a new distribution and select the right S3 buckets as targets, and CloudFront does the rest of the work.

The Workflow

♦  This workflow creates buckets that will be used to store the video content • An API is used to manage VOD content
♦  An input bucket is created to store raw video content uploaded by the content creator
♦  Once a file is uploaded into this bucket, a Lambda function is triggered which creates a MediaConvert job to transcode the video
♦  The transcoded videos are then stored in an output bucket
♦  The API Gateway provides a method for users make requests to retrieve the list of transcoded videos available in the output bucket

Features of the Workflow

1. API that allows the content creator to perform CRUD (Create, Read, Update and Delete) operations on the content
2. Content tagging in the S3 buckets
3. AWS Cognito integration for user management
4. Content Metadata stored in AWS DynamoDB
5. White label branding capability
6. Easy installation with just a single command required to deploy the full stack
7. m3u8 support through Apple HLS

Instructions

Pre-requirements 

AWS CLI 

All the steps require you to have AWS CLI installed. Your credentials need to be set inside ~/.aws/credentials 

S3 Bucket 

A bucket will initially be needed for the Serverless framework. This bucket will be used to store data during deployment. The other buckets will be created during deployment. 

CloudFront Key Pair 

A CloudFront key pair is needed for the pre-signed URLs generation. Instructions are available here. You will need to save the key pair ID and the private key; they will be used in the next section for the configuration. 

NodeJS / NPM 

Instructions are available here to install NodeJS and NPM. 

Serverless 

Instructions are available here to install Serverless. Go to your AWS Console and create an access key for Serverless by following this tutorial. There are details on how to use your access key with Serverless below the tutorial.

Configuration

A configuration file named config.json is available to customize this workflow. Two modifications are mandatory (see below). Once you have made them, you can rename the other values in this configuration file based on your needs. 

Deployment 

You need to set as “deployment bucket” the name of the bucket created above – so Serverless can push content to this bucket for deployment.You can also change the region in which you want to deploy the workflow. 

Distribution 

First, you will need to add information about the key pair generated above into the configuration file: 

• “keyPairId” must be the generated key pair ID 
• “privateKey” is the content of the private key that you have downloaded (Note: Each new line should be replaced by \n, so the private key can be stored as one single line) 

Next, perform the first deployment so Serverless can set up everything required. Once the first deployment is finished, you will need to go to the AWS Console and search for the CloudFront distribution created with this workflow. A base URL will be displayed. This value needs to be stored as “baseURL” in the configuration file. 

Deployment 

• Install required libraries $> npm install
• You can now deploy the workflow with the following command:
  $> serverless deploygt; serverless s3deploy # This command will add events notification support to trigger Lambda for transcodingUsing the APIFirst, you will need to upload a video by generating an upload URL:
  
  Then, once this URL is generated, you can upload your filet to S3:

You can then track the transcoding status by requesting details about your video to the API.

Depending on your video, transcoding might take some time.

Once the transcoding is done, you can get the transcoded output and thumbnails:

Three kinds of content will be available:

• MP4: MP4 version of your video
• HLS: M3U8 playlist files with multiple resolutions (High, Medium & Low)
• Thumbnails: Extracted frames from your video
  If you want to store more details about the video, you can add data into DynamoDB using the API:
  API DocumentationDetailed documentation about the API is available here.

Cognito for User Management

Users are managed using Amazon Cognito which is a service that provides authentication, authorization, and user management for web and mobile applications.

The two primary user pools in this solution are:

• vod-workflow-user-pool-users : For the Users
• vod-workflow-user-pool-admins : For the Admins

A Simple VOD Solution using AWS Elemental Designed For Rapid Deployment

The VOD workflow TrackIt has developed and published is a clean, elegant, and simple solution for any content creator looking to rapidly deploy a VOD solution and requires only a modicum of AWS-related technical expertise. The instructions provided in this whitepaper allow you to quickly and easily assemble your own VOD solution and have it running right away. We have designed this VOD workflow as a turnkey entry-solution that a content creator won’t need to worry about once it’s set up properly.

Please feel free to contact us for any questions, feedback, or suggestions. TrackIt can also offer customization or augmentation of this solution for your particular needs as part of our full-services AWS consultancy and software development practices.