About Amazon Chime SDK

Amazon Chime SDK is a set of tools that organizations can leverage to build their own fully-customizable video conferencing platforms. The Amazon Chime SDK enables developers to make use of the same tools and communications infrastructure as Amazon Chime to add audio calling, video calling, and screen sharing capabilities to their applications.

Creating a Blurred Background on Amazon Chime SDK Applications

One limitation recognized in the Amazon Chime SDK is that, despite being powered by the same communications infrastructure as Amazon Chime, applications using Chime SDK do not have the same ‘Blur My Video Background’ feature that is available in Chime.

aZdzudKEVGkG3ptoyaDoKu2coBFYNR18qxrucZD2Xf6VaI07mInNesmO0MIFVuIdZXAv6vfhxypvx5FOsZKrDK58YxSiFKUdOkr818NIvAMwdNy0YKGfGHcLbgLAORtMk6 0a8DF=s0
‘Blur My Video Background’ feature on Amazon Chime

As a result, users were forced to find other means to blur their backgrounds such as: 

  1. Having a camera with a lens that can handle depth of field. More information here: https://en.wikipedia.org/wiki/Bokeh
  2. Using paid software such as chromacam.meXSplit VCam, or ManyCam. These create a virtual device on the user’s computer. The user sees a new virtual camera device which receives video from the original camera and then processes it frame by frame
  3. Developing custom software that blurs the background
  4. Using a broadcasting software solution like OBS to set up a virtual camera and add blur effects

The new Amazon Chime SDK Video Processing API, launched in December 2020, now provides methods for developers to create blurred backgrounds using the BodyPix model on Tensorflow. The following article will provide readers with the instructions they need to create blurred backgrounds with Amazon Chime SDK on a web browser.

How It Works

Similar to the process of creating a blurred background using paid software such as chromacam.me, the objective with the Video Processing API is to create a new camera device that receives video from the original camera and then converts it frame by frame. 

The DefaultVideoTransformDevice class within Amazon Chime SDK’s Video Processing API allows creation of a new device which will use an inherited class of the VideoFrameProcessor interface in order to modify our video frame by frame.

Video Processing API Browser Compatibilities

IE9fiCMgaypwxNLQM17JL8CSsovI JQoEFJFnuQDyJfd WhNIZepYVE6w5bgozd S7YBlitWsD5K1oTF8d9PR

Guide to the AWS Chime SDK Video Processing API: https://aws.github.io/amazon-chime-sdk-js/modules/videoprocessor.html

About TensorFlow & BodyPix

TensorFlow is an open source machine learning platform that helps developers build and deploy machine learning applications. This tutorial requires users to have at least the 3.8.0 version of TensorFlow. BodyPix is an open-source machine learning model used for real-time person and body-part segmentation in the browser using TensorFlow.js.

Solution Diagram

ukgPHJ9Rbd3LZ FuH0csXshvkA N6e57tmzeorhKuooPKuRBb Vp4mR F6NUg46l rS96SbTOpF szSaaOfhsflMGqnYB8UwszQP lo7LCaE tSjMmBRYOLhe78IRFDa5oqHi3AV=s0
Amazon Chime SDK Blurred Background Workflow


The following tutorial shows readers how to create a blurred background on the browser demo of Amazon Chime SDK, available here: https://github.com/aws/amazon-chime-sdk-js/tree/master/demos/browser

npm links:

GitHub links:

To begin, we have to install TensorFlow and the Bodypix trained model from TensorFlow.

While there are multiple APIs and backends/platforms in the @tensorflow/tfjs package, we will not be using any of these APIs or specific platforms in this tutorial. 

Side note for readers: This npm package could be used with node.js, React Native, and other frameworks.

Tensorflow.js Packages

For the backend, we will be using WebGL because it is the fastest processing backend choice for the BodyPix model.

lVv00 HJkR7l2GHsSCiaYvxCPM JY3 pSD45FAzqmp 7mRkTvjt3mY3tHOskFw4oFNlmGsVpJEhATLhAE663CivtFaen8fK6DQwcpIR7Mbuf3 qnt34HQgdp6CEaFLM9h9Z2b58o=s0

Let’s start by installing the TensorFlow and the BodyPix packages:

aNKa0wMgJAO7HRNq8T9c5mlTmgb16hrIrDedPAQ PTbA81Mk5bNzJLODYe YLyEwdO18NFe f96nZ4daL95lu8dK0
W dfsVrNUL6I8xo2ZXryEo R7CLni67ykCoSnW34kR8CHwWKF59hWQ4WkogpXKVJ6aFQNIkwC v 72X6Z82NSoN5Gp0eZiiC7lVl9KOLm6bfOL1e8TeTNZKl80IKYG AD5W4Tswb=s0

You must use @tensorflow/tfjs version greater or equal than 3.8.0.

Next, we import these modules in the meeting script:

fGWbbHUFAfS2w6LjNu2cSBLAtioNbk0sBwHtcYL4 tMWv4uRh5W08TTD6Sy5QPQBOApxL9 AtB81eLai4oOB0bamzYWxaK5xOXs0nYOkuqeyA yOGWOFb9FanEGIhSrXDXxjL tV=s0

We have to set the backend to ‘webgl’:

gWp8RJRkEXsdZ3EGe7Wv5swAZKNqVQ jRFDuA5FrrTYDetbco9TBlkJFa3CKPtmUUOWC9WCs2VBFo1reTuZG06 qg6 PbzidLdk3fnD5j 4Yx4X5JxgmFU 9kBAbYAY4Rm1uPm5l=s0

Then we have to load the bodyPix model:

From the body-pix model’s README.md

Some information about the configuration params in bodyPix.load():

LKOoZOL7CWettI4aHyiIIi5icnvQnFjg4HNaI4eHTaQdmukTELYYU7BuyW7p7r gv ZOZI TOAHkQEu5S Lb2ilMUhlz9HqoBU57N6s0i39jF2 KDatOtlTeu 33k84zfEcVfaEg=s0
From the body-pix model’s README.md

Now we create our class to process the video frame-by-frame. This class must implement the VideoFrameProcessor interface.

eiIhk2aAP2cr1qipiqaLS3rOzS607PDVY spNEyP1JgjHj8rSwDodot1gube5QqdQdvuB89qUiCXKJRj7oCmmW0YihZnlG4 sBNhJ74QJNGeiZ4UAFowhoKHmzStaAp0VGLMDGqy=s0
Blurred Background Frame-by-Frame Processing Class

The last thing we have to do is to create a new device with the previous class we just created above.

We use the new device with audioVideoFacade::chooseVideoInputDevice method.

More information about DefaultVideoTransformDevice here:



Result without headset:

hNFwAmzYJPievLY yLqc6DLjeGDLvHvWKlQYLhLLH8vw9KLlWjbisrtPuOkJ8JloDHDf6SQMckJm8fUAop8MFM7MlwgHCDUGigTbmY2TvCBCPe2VpkNCtIBUXVo5bIss4 PsJ1GU=s0
Blurred background without headset

Result with headset:

7KWE9qiPl9orqFfMJ0OOVXk4ztsuq9gzNsDGDACkd5vx 8Py0jjQB1QDlZHH1hKDYHHVv R2GhH GcDDTmhhgF PqPeC4iH8R dqx2CDfqMtR1FmpcwdaTBeLRLD nwG6bY6dbtu=s0
Blurred background with headset

The code is available here: https://github.com/trackit/amazon-chime-sdk-js/tree/master/demos/browser.

Forked from: https://github.com/aws/amazon-chime-sdk-js/tree/master/demos/browser.

About TrackIt

TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Venice, CA.

TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.

TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build, and deploy customized solutions tailored to your exact requirements.

In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.