Contents
About Amazon Chime SDK
Amazon Chime SDK is a set of tools that organizations can leverage to build their own fully-customizable video conferencing platforms. The Amazon Chime SDK enables developers to make use of the same tools and communications infrastructure as Amazon Chime to add audio calling, video calling, and screen sharing capabilities to their applications.
Creating a Blurred Background on Amazon Chime SDK Applications
One limitation recognized in the Amazon Chime SDK is that, despite being powered by the same communications infrastructure as Amazon Chime, applications using Chime SDK do not have the same ‘Blur My Video Background’ feature that is available in Chime.
As a result, users were forced to find other means to blur their backgrounds such as:
- Having a camera with a lens that can handle depth of field. More information here: https://en.wikipedia.org/wiki/Bokeh
- Using paid software such as chromacam.me, XSplit VCam, or ManyCam. These create a virtual device on the user’s computer. The user sees a new virtual camera device which receives video from the original camera and then processes it frame by frame
- Developing custom software that blurs the background
- Using a broadcasting software solution like OBS to set up a virtual camera and add blur effects
The new Amazon Chime SDK Video Processing API, launched in December 2020, now provides methods for developers to create blurred backgrounds using the BodyPix model on Tensorflow. The following article will provide readers with the instructions they need to create blurred backgrounds with Amazon Chime SDK on a web browser.
How It Works
Similar to the process of creating a blurred background using paid software such as chromacam.me, the objective with the Video Processing API is to create a new camera device that receives video from the original camera and then converts it frame by frame.
The DefaultVideoTransformDevice class within Amazon Chime SDK’s Video Processing API allows creation of a new device which will use an inherited class of the VideoFrameProcessor interface in order to modify our video frame by frame.
Video Processing API Browser Compatibilities
Guide to the AWS Chime SDK Video Processing API: https://aws.github.io/amazon-chime-sdk-js/modules/videoprocessor.html
About TensorFlow & BodyPix
TensorFlow is an open source machine learning platform that helps developers build and deploy machine learning applications. This tutorial requires users to have at least the 3.8.0 version of TensorFlow. BodyPix is an open-source machine learning model used for real-time person and body-part segmentation in the browser using TensorFlow.js.
Solution Diagram
Tutorial
The following tutorial shows readers how to create a blurred background on the browser demo of Amazon Chime SDK, available here: https://github.com/aws/amazon-chime-sdk-js/tree/master/demos/browser
npm links:
- https://www.npmjs.com/package/@tensorflow/tfjs
- https://www.npmjs.com/package/@tensorflow-models/body-pix
GitHub links:
- https://github.com/tensorflow/tfjs#readme
- https://github.com/tensorflow/tfjs-models/tree/master/body-pix
To begin, we have to install TensorFlow and the Bodypix trained model from TensorFlow.
While there are multiple APIs and backends/platforms in the @tensorflow/tfjs package, we will not be using any of these APIs or specific platforms in this tutorial.
Side note for readers: This npm package could be used with node.js, React Native, and other frameworks.
For the backend, we will be using WebGL because it is the fastest processing backend choice for the BodyPix model.
Let’s start by installing the TensorFlow and the BodyPix packages:
You must use @tensorflow/tfjs version greater or equal than 3.8.0.
Next, we import these modules in the meeting script:
We have to set the backend to ‘webgl’:
Then we have to load the bodyPix model:
Some information about the configuration params in bodyPix.load():
Now we create our class to process the video frame-by-frame. This class must implement the VideoFrameProcessor interface.
The last thing we have to do is to create a new device with the previous class we just created above.
We use the new device with audioVideoFacade::chooseVideoInputDevice method.
More information about DefaultVideoTransformDevice here:
Results:
Result without headset:
Result with headset:
The code is available here: https://github.com/trackit/amazon-chime-sdk-js/tree/master/demos/browser.
Forked from: https://github.com/aws/amazon-chime-sdk-js/tree/master/demos/browser.
About TrackIt
TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Venice, CA.
TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.
TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build, and deploy customized solutions tailored to your exact requirements.
In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.