OMTG-4B Beats Gemini 2.5 Pro by 15 Points at Finding Every Video Event
A new benchmark, 56k-sample dataset, and GRPO-trained model tackle the overlooked problem of finding every instance of a repeated action in video
- New task formalized: One-to-Many Temporal Grounding (OMTG) requires finding all disjoint video segments matching a query, not just one.
- Existing models fail completely: Qwen2.5-VL series scores 0% Count Accuracy; even Gemini 2.5 Pro and Seed-1.8 are in the "weak capability" zone.
- New benchmark and metrics: 340-sample OMTG Bench with Count Accuracy (C-Acc) and Effective Temporal F1 (EtF1) that zero-out scores when event counts are wrong.
- 56k training dataset: Built via a 5-stage automated pipeline using Qwen3-VL-235B and Gemini 2.5 Pro for discovery, grounding, verification, and dense captioning.
- OMTG-4B beats all baselines: Achieves 43.65% EtF1, outperforming Gemini 2.5 Pro by 15.85 points using a GRPO-trained 4B model.
- Bonus: one-to-one grounding also improves -- RL training on OMTG data alone boosts standard single-segment grounding benchmarks, suggesting the task builds generalizable temporal reasoning.
Ask any video model to find every time a basketball player shoots a three-pointer in a 20-minute game, and it will almost certainly give you just one clip. That is the gap this paper targets. Researchers have published a paper formalizing One-to-Many Temporal Grounding (OMTG) -- the task of localizing all disjoint video segments that match a single text query -- and delivering a complete solution: a new benchmark, a curated training dataset, and a model that beats Gemini 2.5 Pro by more than 15 points on the new metric.
The blind spot in video understanding
Video temporal grounding has seen significant advances driven by multimodal large language models, which are gradually surpassing traditional fine-tuned methods. But there is a structural assumption baked into almost every existing approach: one query maps to one segment. Prior research predominantly focuses on single-segment retrieval, yet real-world scenarios often require localizing multiple disjoint segments for a single query.
Think of the kinds of queries that actually matter in practice: "the chef adds salt," "the presenter clicks to the next slide," "the car changes lanes." In any non-trivial video, these events recur. The benchmark reveals a critical capability gap: standard open-source models such as the Qwen2.5-VL series yield 0% Count Accuracy, failing to capture the one-to-many complexity. Even frontier proprietary models like Gemini 2.5 Pro and Seed-1.8 score in the "weak OMTG capability" zone.
Why existing metrics fail here
Standard temporal grounding uses Recall@1 with a temporal IoU (tIoU) threshold: did the model find the right segment? In the one-to-many setting, that metric becomes actively misleading. A model that merges two distinct events separated by a short gap into one continuous span can score a tIoU of 0.9 -- technically "correct" -- while completely missing the fact that two separate events occurred.
The paper introduces two new metrics to fix this:
- Count Accuracy (C-Acc):
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
