Nanjing University's TempR1 Teaches One Video AI to Master All Five Timestamp Tasks
A new multi-task reinforcement learning recipe teaches video language models to pinpoint events in time across five different temporal tasks at once.
ProTakeaways
- TempR1 trains one MLLM on five temporal video tasks jointly using GRPO reinforcement learning.
- Three custom localization rewards handle one-to-one, many-to-one, and many-to-many prediction-to-ground-truth mappings.
- Built on Qwen2.5-VL-7B with a 60K sample multi-task corpus curated across public datasets.
- Hits 71.0 mF1 on ActivityNet TAL, beating MUSEG by 13 points, and 71.1 mIoU on QVHighlights.
- Ablations confirm strong cross-task synergy; adding tasks lifts individual benchmark scores.
- RL fine-tuning preserves general video understanding on VideoMME and MVBench, unlike supervised fine-tuning.
Video language models are getting fluent at describing what happens on screen, but ask them exactly when something happens and things fall apart quickly. A new framework called TempR1, from researchers at Nanjing University, ByteDance, and Shanghai AI Lab, trains a single MLLM to handle five time-related video tasks simultaneously using reinforcement learning with rewards tailored to each task's temporal structure.
Built on Qwen2.5-VL-7B and trained on roughly 60K curated samples spanning temporal grounding, dense temporal grounding, temporal action localization, video highlight detection, and grounded video QA, TempR1 posts state-of-the-art results across the board, including a 13-point jump in mF1 over MUSEG on ActivityNet-v1.3 action localization.
Why timestamps break MLLMs
Most current video MLLMs are trained on general video-text alignment and struggle to output precise time intervals. A handful of recent reinforcement-learning approaches have improved this, but they mostly focus on static images or short clips, with limited exploration of long-form temporal reasoning beyond basic grounding. That single-task tunnel vision produces models that can localize one described event but choke when asked to enumerate every instance of an action or provide temporal evidence for a QA answer.
Supervised fine-tuning has its own trap. Rigid token-level supervision on temporal instruction data tends to overfit and erodes the base model's general reasoning ability. RL-based fine-tuning, which rewards any temporally and semantically plausible prediction rather than one exact string, has emerged as the better path, though the community has been building one narrow reward per task.