Social Department Case Study: Enabling Multimodal AI Content Discovery Workflows

Customer Challenge

Social Department engaged TrackIt to support the development of its AI-powered Content Analysis and Discovery (AICAD™) platform designed to support semantic analysis and content discovery across video content using multimodal AI workflows.

The customer needed to accelerate delivery of a functional MVP within a tight product launch timeline while ensuring the platform was stable, performant, and ready for demonstration. The solution required semantic analysis across visual, audio, transcript, and scene embeddings to power multiple content discovery workflows capable of identifying high-value moments from video assets.

Close collaboration between engineering teams was also required to configure and tune the underlying AI pipeline, validate functionality against defined acceptance criteria, and establish a scalable foundation for future AI-driven marketing and promotional workflows.

Implementation

TrackIt partnered directly with Social Department by augmenting its engineering team with AI/ML and cloud engineering expertise focused on accelerating delivery of the AICAD™ MVP.

Development, configuration, and tuning efforts focused on a multimodal video pipeline built on TwelveLabs and AWS services including Amazon Bedrock. Semantic analysis across visual, audio, transcript, and scene-level embeddings enabled the platform to surface contextually relevant moments from video content.

AICAD™ Pipeline Overview

The AICAD™ pipeline leverages AWS Step Functions to orchestrate ingestion, processing, enrichment, and semantic indexing workflows across video assets. Uploaded content is transcoded through AWS Elemental MediaConvert, segmented into short clips, and processed through multiple AI and ML services including TwelveLabs Marengo, Amazon Rekognition, Amazon Bedrock, and Claude-based enrichment workflows.

The pipeline generates multimodal embeddings and structured metadata spanning visual, audio, transcript, scene, OCR, and character-level analysis, enabling hybrid semantic and keyword-based retrieval across the indexed video corpus. Search and ranking workflows combine vector search, reranking, and AI-driven query expansion to surface contextually relevant clips across multiple content discovery experiences.

TrackIt supported the development, configuration, and tuning of the pipeline to help surface contextually relevant clips and moments across multiple content discovery experiences.

Content Discovery Workflows

Five core content discovery workflows were developed and tuned within the MVP, including:

• Shareable scenes
• Quotable quotes
• Memeable moments
• The hook
• Trendable moments

 
Shareable Scenes: When a creator hits the Shareable Scenes button, the platform scans campaign content to identify clips with strong social sharing potential. A dedicated worker retrieves campaign assets from the database, resolves video content through SDAP, and executes a two-stage AI pipeline.

AICAD™ performs a parallel vector search across the content library to surface candidate clips, while Amazon Bedrock ranks the results based on factors such as emotional resonance, visual distinctiveness, and narrative completeness. The final output is returned to the UI as a ranked list of scenes containing clip IDs, timestamp ranges, and scene labels ready for editing and publishing workflows.

Quotable Quotes: When triggered, the Quotable Quotes workflow identifies dialogue moments with strong standalone and social engagement potential. The worker retrieves campaign content and executes an AICAD™ semantic search to identify scenes containing notable dialogue moments.

Amazon Bedrock then ranks candidate clips based on factors such as wit, cultural resonance, and contextual clarity. A secondary AICAD™ retrieval step fetches the precise transcript associated with each shortlisted scene to ensure quote accuracy. The resulting output includes speaker attribution, quote text, and timestamp ranges for downstream content and captioning workflows.

The Hook: When a creator uses The Hook workflow, the platform identifies moments designed to capture viewer attention within the opening seconds of a clip. The pipeline executes semantic searches across campaign assets before processing candidate clips through a Bedrock scoring model. 

The scoring workflow generates an overall hookScore alongside supporting sub-scores evaluating factors such as intrigue, visual intensity, and narrative tension. This additional scoring context helps social and editorial teams select cold opens, thumbnails, and attention-grabbing moments for promotional edits.

Memeable Moments: When the Memeable Moments workflow is triggered, the platform identifies clips with strong meme potential using semantic search and AI-generated captioning. The pipeline uses AICAD™ search and Amazon Bedrock captioning capabilities to rank candidate moments before extracting high-quality still frames from selected clips through AICAD™ frame extraction services.

The workflow then generates completed meme assets with text overlays and stores the rendered PNG outputs in Amazon S3. Once processing completes, the UI receives fully rendered meme assets through signed URLs alongside the associated clip references.

Trend Align: Trend Align connects a campaign’s existing content to what is actually trending, rather than asking teams to manually identify the overlap. The pipeline runs a multi-step Amazon Bedrock workflow that transforms trend signals into search vectors, processes them through AICAD™ semantic searches to identify matching clips, reruns reranking passes to refine relevance, and generates captions aligned to the associated trend context.

Each result includes a relevanceScore, associated trend metadata, a ready-to-use caption, and a sourceMode flag indicating whether the trend data was retrieved live or served from cache. The final output surfaces a shortlist of existing campaign clips paired with generated captions and contextual explanations describing how each clip aligns with the current cultural moment.

In parallel, functionality validation and performance testing helped ensure the platform met operational and demonstration requirements.

The underlying pipeline architecture also established the foundation for additional AI-powered promotional and marketing intelligence workflows.

Outcome

Through close collaboration between engineering teams, Social Department successfully accelerated the delivery of its AI-powered content discovery platform within the required launch timeline.

The engagement resulted in a functional MVP capable of identifying and surfacing meaningful video moments across multiple discovery workflows. By contributing AI/ML pipeline development, tuning, and validation expertise, TrackIt helped support product readiness while enabling Social Department to continue expanding its AI-driven media analysis capabilities.