Skip to main content
Back

NVIDIA's VideoITG Picks Smarter Frames so Video AI Stops Missing the Point

NVIDIA's VideoITG teaches video language models to pick frames based on the question being asked, matching 64-frame accuracy with just 16.

Pro

  • NVIDIA's VideoITG selects video frames based on user instructions, not fixed intervals, boosting Video-LLM accuracy across benchmarks.
  • The CVPR 2026 Highlight paper ships a 40K-video, 500K-annotation dataset built by the VidThinker pipeline.
  • 16 VideoITG-selected frames match 64 uniform frames on VideoMME; 32 frames beat 64-frame baselines consistently.
  • Plug-and-play across LLaVA-Video, InternVL2.5, InternVL3.5, Qwen3-VL, Eagle2.5; 8B model beats 26B baseline with selection.
  • Three selector variants tested; full-attention pooling-based classification wins over causal-attention and text-generation designs.
  • Code, weights, and evaluation scripts released Apache 2.0 at github.com/NVlabs/VideoITG.

Video language models keep hitting the same wall: they can only look at a handful of frames from a long video, and picking the wrong ones tanks accuracy. An NVIDIA-led project called VideoITG attacks that bottleneck directly by turning frame selection into an instruction-following task, with the model choosing which moments to look at based on what the user actually asked.

The paper, VideoITG: Multimodal Video Understanding with Instructed Temporal Grounding, was selected as a CVPR 2026 Highlight. It introduces a framework that adapts frame sampling to user instructions, backed by an automated annotation pipeline called VidThinker that generates instruction-conditioned clip captions, retrieves relevant video segments, and performs fine-grained frame localization. The resulting VideoITG-40K dataset contains 40K videos and 500K temporal grounding annotations, and the plug-and-play model improves accuracy across multimodal video benchmarks.

Why uniform sampling keeps failing

Most Video-LLMs feed a fixed number of evenly spaced frames into the model, ignoring what the question is about. If someone asks about a person's shoes during a two-minute clip, uniform sampling might grab 32 frames of scenery and miss the one moment that shows the shoes. Prior fixes either compress features (pooling, similarity pruning, clustering) or extend context length, but both hurt precision when a query hinges on a specific moment.

Instruction-aware selectors like SeViLA, AKS, Q-Frame, and Frame-Voyager exist, but the authors argue they still fall short on multi-event or time-sensitive queries because they treat frames largely in isolation. VideoITG is designed to handle three regimes at once: locating scattered temporal cues, sampling densely through motion-heavy segments, and keeping broad coverage for open-ended prompts.