Horizon Robotics' Progress-Think Teaches Robots to Follow Complex Instructions Step by Step
A new three-stage training framework teaches navigation agents to track their semantic position within multi-step instructions, hitting state-of-the-art on R2R-CE and RxR-CE.
ProTakeaways
- Progress-Think introduces semantic progress reasoning that predicts which instruction span an agent has already completed, replacing scalar progress signals.
- Hits 60.1 SR and 53.6 SPL on R2R-CE Val-Unseen using only monocular RGB, beating depth and panorama methods.
- Uses a three-stage pipeline: self-supervised prefix alignment, progress-guided policy pretraining, and GRPO co-finetuning.
- Trained with zero external data beyond the simulator, no human or GPT-4 annotations of intermediate steps needed.
- Two 2B NVILA models run at 36.6s per episode on a 4090, faster than 8B baselines like NaVILA.
- Project page is live; the paper reports results on R2R-CE and RxR-CE benchmarks.
Getting a robot to follow a sentence like "go past the kitchen, turn left at the hallway, and stop by the bed" is deceptively hard. The agent has to know not just where it is in space, but which clause of the instruction it is currently working on. A team from Renmin University and Horizon Robotics argues that this second piece is exactly what modern vision-language-action (VLA) navigation models miss, and their new system Progress-Think shows what happens when you build it in explicitly.
Progress-Think reframes progress as the actual span of instruction text the agent has already fulfilled, rather than a percentage or a distance-to-goal. That semantic reframing, paired with a self-supervised training recipe, delivers state-of-the-art results without any external reasoning annotations.
Why progress tracking has been broken
Vision-Language Navigation in Continuous Environments (VLN-CE) drops an agent into a photorealistic 3D scene and gives it a natural-language route to follow using only a monocular RGB camera. Modern VLA navigation frameworks typically rely on end-to-end policy learning, where any notion of progress is implicitly buried within action prediction without an explicit representation. Traditional approaches, by contrast, approximate progress using numeric measures such as geometric ratios or remaining distance.
The authors point out that both approaches miss a structural fact about the task: visual observations and instruction semantics exhibit monotonic co-progression, meaning as observations accumulate, the aligned instruction prefix extends monotonically over time, with later progress consistently building on earlier progress. A scalar can tell you that you have covered 40% of the path, but not whether you have already turned at the hallway.