Skip to main content
Back

Amazon's ViLL-E Kills the Two-Model Video Pipeline With One 2.5B System

Amazon researchers unify video generation and embedding tasks in a single model, closing the retrieval gap that has plagued VideoLLMs against specialized dual-encoder systems.

Pro

Paper, Full Access
  • ViLL-E unifies video generation and embedding tasks in one model, closing the retrieval gap for VideoLLMs. Paper
  • Adaptive embedding head lets the model spend more inference tokens on complex videos, fewer on simple ones.
  • Built on PaliGemma-3B with a novel KV-Former pooling design using 256 learnable tokens.
  • Three-stage training: 10M Shutterstock pairs, 200K Claude-recaptioned clips, then multi-task finetuning across four tasks.
  • Beats VideoLLMs by 7%+ on temporal localization and matches fine-tuned dual-encoders on MSR-VTT and DiDeMo.
  • Zero-shot gains of 5% on composed video retrieval and 11% on long-caption search versus prior state of the art.

Video large language models have carved out a comfortable niche answering questions and captioning clips, but they consistently lose to smaller specialized systems the moment you ask them to actually find something. A new architecture called ViLL-E, from researchers at Amazon and the University of Central Florida, tries to fix that split by teaching a single VideoLLM to produce both fluent text and discriminative embeddings, with a twist: the model decides on its own how many tokens to spend embedding a given video.

Two stacks, one job

The retrieval gap is real and measurable. VideoLLMs like Qwen2.5-VL and LLaVA-ST handle QA and captioning well, but every top performer in video retrieval, temporal localization, or cross-modal search (VidLA, QD-DETR, SigLIP) gets there with embeddings, a capability generative VideoLLMs simply lack. Teams building video search or moment retrieval pipelines have been forced to maintain two entirely different model stacks, one for reasoning tasks and another for matching.

Recent NLP work such as GRIT and E5 showed that a pre-trained LLM can be turned into a strong retriever with a surprisingly small amount of contrastive finetuning, which raised an obvious question for the video world: could one model do both? ViLL-E is the video-side answer.

An embedding head that knows when to stop

The architecture starts with PaliGemma-3B as the multimodal backbone and bolts on what the authors call a KV-Former embedding head. It uses the LLM's own tokens as queries against a set of learnable key and value pooling tokens, acting as a bottleneck akin to dictionary learning while still allowing variable-length outputs. Ablations landed on 256 pooling tokens as the sweet spot, well above the 32 queries typical for Q-Former designs.