Written by Tiago Valenca, Software Engineer, and Mateus Alexandre, DevOps Engineer
Finding a specific video in a vast content collection is a common challenge for modern media enterprises, particularly when relying on traditional search methods that depend on metadata such as titles and descriptions. This issue is especially prevalent in the sports genre, where the sheer volume and variety of content make it even more difficult. As a result, users often struggle to locate the exact footage they need, leading to frustration and inefficiency.
Contents
AI-Driven Video Library Natural Language Search
TrackIt recently showcased a new integration of its Trackflix product with an AI model, enabling natural language search capabilities. This integration was powered by an API built on Amazon API Gateway, utilizing additional AWS services such as Amazon DynamoDB, AWS Lambda, and Amazon S3.
The Challenge
Trackflix enables customers to launch their own Content Management System (CMS) and deliver videos with a Netflix-like experience. Traditionally, video search has been limited by titles, descriptions, and other metadata. However, as content volume grows, users struggle to intuitively find what they are searching for. Traditional keyword searches have proven insufficient to meet the demand for a more user-friendly search experience, but this changes with generative AI models. For this project, TrackIt chose to use a Twelve Labs model called Marengo due to its easy-to-integrate API that allows the handling of videos directly from an AWS environment.
The Process
- Identifying the Need: Our customers highlighted the limitations of traditional search methods, requiring a solution that could understand natural language queries and deliver relevant video results based on content, not just metadata.
- Choosing the Right Partner: Twelve Labs was chosen for its advanced AI models specializing in multimodal AI-powered video analysis technology that combines visual, audio, and textual data to extract meaningful insights. Their expertise and technology aligned perfectly with the project requirements.
- Building the Solution:
- API Development on AWS: An API was developed using Amazon API Gateway to facilitate seamless communication between Trackflix and Marengo, the Twelve Labs AI model, creating a robust and scalable API efficiently.
- Lambda Functions: AWS Lambda functions were utilized to process search queries and interact with the AI model, ensuring a cost-effective and scalable serverless approach.
- DynamoDB: DynamoDB was used to save data from the videos indexed in Twelve Labs, enabling integration with Trackflix.
- Testing and Refinement: Extensive testing ensured a smooth integration, with search results being accurate and relevant.
Results
As demonstrated in the above video, it is now possible to search for videos based on their content, not just metadata. However, some limitations still exist. For example, depending on the prompt used, the results may not always match the quality of those shown, and the video length for indexing is currently limited to 20 minutes (a limitation of the Twelve Labs free plan). Additionally, the processing time is currently 1:1, meaning a 20-minute video takes 20 minutes to analyze. Both of these limitations are expected to improve significantly with the paid plan and further development as AI models continue to evolve.
Closing Thoughts
By integrating Trackflix with a generative AI model via a robust API on AWS, a new standard for natural language search has been established. Future enhancements will continue to incorporate similar cutting-edge solutions.
TrackIt is currently applying generative AI technology across numerous customer workloads, optimizing various operations and driving innovation. This approach aims to assist media, entertainment, and gaming companies in leveraging AI for enhanced content discovery, production efficiency, and audience engagement.
About TrackIt
TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.
We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.
Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.