Highlights in video content refer to the most significant, interesting, or impactful moments within a video. In sports, highlights capture key plays such as goals, winning shots, crucial saves, and other memorable events that are essential to the game’s excitement and outcome. These moments are often what fans anticipate most, as they encapsulate the match’s most thrilling parts.
When a sports match concludes, news outlets worldwide race to generate and share the best highlights as quickly as possible, ensuring fans can relive the excitement almost immediately. This rapid turnaround is crucial for keeping audiences engaged and informed. Efficient editing processes are also essential, allowing editors to quickly compile and enhance highlight reels, delivering polished content in record time.
Contents
Leveraging AI and AWS for Seamless Sports Video Highlighting
TrackIt tested a cutting-edge integration of Pegasus, an AI model from Twelve Labs, designed to automatically index videos and generate highlights. Two demos were conducted to verify Pegasus’s effectiveness and explore its potential use cases in identifying and extracting the most significant moments from sports videos. By leveraging AWS services such as DynamoDB, Lambda Functions, S3, AWS Elemental MediaConvert, and API Gateway, integrations were developed to enable users to obtain highlight content without manual editing. Additionally, the process generates an XML from the highlights for seamless import into Adobe Premiere for NLE editing.
First Demo: Testing Twelve Labs AI for Sports Video Highlights
Objective
The demo aimed to evaluate the effectiveness of the Twelve Labs Generate API in identifying highlights from sports matches, specifically soccer and League of Legends. The objective was to develop a system that, based on a given prompt, can generate highlights from a specific video and compile them into a new video containing only the highlights.
The Process
Import Workflow
- Content Upload: Content is imported into an S3 bucket, which triggers a Lambda function to request the Twelve Labs API for indexing the content.
- Metadata Storage: Twelve Labs returns metadata of the video, which is uploaded into a DynamoDB table.
Highlight Generation Workflow
- Metadata Retrieval: A Lambda function retrieves the information from DynamoDB and makes a request to the Twelve Labs API to generate the highlights.
- Highlight Generation: Twelve Labs returns the highlights to Lambda, which creates a job in MediaConvert to generate the video from the highlights.
- Storage: The highlights video is stored in S3.
The Results
Pegasus by Twelve Labs was highly effective in identifying the highlights of the sports match, accurately capturing key moments. However, some instances revealed that the highlights were not perfectly precise in terms of their starting and ending points, indicating room for improvement in fine-tuning the model for more accurate segmentation of the highlights.
Second Demo: Optimizing Video Editing Processes with AI and XML Workflows
Objective
The second demo aimed to refine the use of the Generate API by shifting from automatic video generation to assisting editors during the editing process. The plan was to create a front-end application where users can select a video, choose a prompt, and receive an XML file containing the highlights generated by Twelve Labs. These highlights can then be imported into Adobe Premiere for further editing, enhancing the editor’s workflow.
The Process
Import Workflow
This workflow is similar to the previous demo. The video is imported into an S3 bucket and indexed by Twelve Labs, with the metadata stored in DynamoDB.
Highlight Generation (XML) Workflow
Front-End: The front-end application features two main screens. The first screen displays the videos stored in the database, all of which are indexed in Twelve Labs for generating highlights. The second screen allows users to send prompts to Twelve Labs and receive the resulting XML for import into Adobe Premiere for non-linear editing (NLE). The application is hosted on Amazon CloudFront.
Back-End (Serverless): The back-end leverages API Gateway and two Lambda functions: one for interfacing with the Twelve Labs API to generate highlights and produce XML, and another for retrieving video information from DynamoDB. The highlight generation process with Twelve Labs is similar to the previous demo, with the key difference being the exclusion of MediaConvert from this workflow.
The Results
The results of the demo were satisfactory, as the XML file was successfully created and imported into Adobe, allowing the content’s highlights to be viewed. In the initial demo, it was noted that Pegasus is not yet entirely precise in identifying the entry and exit points of a highlight, indicating that generating a fully automated video might not be the optimal use of this technology.
However, this demo showcased a more effective application: assisting editors in selecting the best moments of a match. While the model can identify highlights, it still requires refinement in pinpointing exact moments—a task that editors can efficiently accomplish using Adobe and XML. This combination of AI and editorial expertise leads to a more accurate and practical solution for video editing.
Closing Thoughts
The demos conducted with Pegasus, the Twelve Labs AI model highlight the potential of AI-driven tools to enhance the video editing process, particularly in the sports domain. While the model shows promise in automatically identifying key moments, further refinement is needed to achieve precise results. The shift toward assisting editors rather than fully automating the process proves to be a more effective application of the technology.
By combining AI capabilities with editorial expertise, a more accurate and efficient workflow can be achieved. TrackIt remains committed to exploring and implementing innovative solutions that empower media professionals to deliver high-quality content with greater speed and precision.
About TrackIt
TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.
We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.
Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.