Skip to main content
Back

Tsinghua's R²VLM Beats GPT-5 at Tracking Robot Task Progress in Video

A new vision-language model tracks how far along an embodied agent is by reasoning over short video clips and an evolving chain of thought.

Pro

  • R²VLM estimates embodied task progress by reasoning recursively over short video snippets with an evolving chain of thought as memory.
  • Built on Qwen2.5-VL-7B, trained via SFT plus PPO with a five-part reward including a cross-turn improvement signal.
  • Hits 2.19 progress MAE and 0.988 accuracy on ALFRED, beating GPT-5, Gemini-2.5-Pro, and Qwen2.5-VL-72B.
  • Snippet input keeps inference time constant while full-video baselines grow roughly linearly with video length.
  • Provides dense rewards that lift SPRINT online RL from 2.20 to 2.45 average completed steps on multi-step tasks.
  • Models, datasets, and benchmarks released on HuggingFace.

Tracking how much of a task an agent has actually finished sounds simple until you try to do it from raw video. Was the coffee mug picked up, or just nudged? Did the robot put the meat in the fridge, or is it still holding it? A team from Renmin University of China, Tsinghua, and RayNeo.AI has released R²VLM, a vision-language model that estimates task progress by reasoning recursively over short video snippets while keeping a running chain of thought that acts as memory.

Instead of stuffing an entire long video into a VLM's context, R²VLM processes it in pieces and forces the model to write down what it has seen, what's left, and how far along the task is. That written trace becomes the input for the next iteration, so global context survives even when the visual input is local.

Why progress estimation is harder than it looks

Recent work like GVL and ROVER has explored VLMs for task progress estimation, but those approaches lean on video understanding and large context windows while leaving the reasoning potential underexplored. The authors point to two problems that have blocked practical deployment:

  • Embodied tasks often consist of multiple subtasks with complex temporal dependencies, and a model must reason about these dependencies and align them with visual observations.
  • Long video trajectories incur substantial computational and time costs when processed by VLMs, rendering such approaches impractical for real-world deployment.

Earlier feature-distance methods like LIV compared the current frame to a goal image, which ignores the temporal structure of a multi-step task. Video-based approaches like GVL and ROVER used in-context prompting on frozen VLMs, but leaned on raw video understanding rather than reasoning.