Skip to main content
Back

HKUST's VP-VLA Beats GR00T and π0.5 by Drawing on Robot Camera Images

A dual-system approach draws crosshairs and boxes on camera feeds to tell robots where to act, boosting success rates by up to 8.3%.

Pro

  • VP-VLA overlays crosshairs on target objects and bounding boxes on destinations to guide robotic manipulation policies.
  • A Qwen3-VL-4B System 2 planner triggers on gripper-state changes, using SAM3 for precise object localization.
  • Improves Robocasa-GR1-Tabletop success by 5% (53.8%) and SimplerEnv by 8.3% (58.3%) over QwenOFT baseline.
  • Auxiliary grounding loss forces the VLM to predict prompt coordinates as discretized JSON, preventing overlays being treated as noise.
  • Real-world OOD color test: 75% success versus 29.2% for the baseline, showing strong attribute generalization.
  • Code available on GitHub; limited to 2D primitives, adds ~0.78s planner latency per event.

Robot policies that translate a spoken instruction directly into motor commands have become the default recipe for general-purpose manipulation. One neural network has to interpret language, locate the right object in a cluttered scene, and produce smooth motor commands, all in a single forward pass. A new paper called VP-VLA from HKUST, CUHK, and SmartMore argues that much of that pain goes away if you just draw on the image before feeding it to the policy.

The idea is disarmingly simple. A high-level planner figures out what to grab and where to put it, then overlays a crosshair on the target object and a bounding box on the destination in the RGB image. A separate low-level controller then follows those visual anchors. On the Robocasa-GR1-Tabletop benchmark, VP-VLA improves the average success rate by 5% over the baseline, surpassing GR00T-N1.6 without additional large-scale robotic pretraining. On SimplerEnv it achieves an 8.3% absolute improvement, beating prior VLA models including π0.5.

Why monolithic VLAs keep tripping over themselves

Vision-Language-Action (VLA) models fine-tune a pretrained vision-language model on robot demonstrations so it can output continuous control signals. That works surprisingly well in-distribution, but recent findings show that substituting meaningful language with gibberish barely affects performance, meaning the policy is often not actually grounding the instruction in the scene. When a novel object shows up, or an object lands in a spatial configuration the model never saw, precision collapses.

Earlier attempts to fix this bolted on intermediate representations like goal images, predicted trajectories, or dense segmentation masks. Because those intermediate modalities are not natively understood by standard VLA models, they typically require task-specific supervision and complex end-to-end training that can compromise the model's inherent reasoning capabilities. Dense masks in particular clash with the RGB inputs the backbone was pretrained on, forcing extra encoders and fusion layers.