VGA Beats π0.5 by Teaching Robots to See in 3D
A new robot policy swaps the language-model brain for a 3D geometry backbone, beating pi-0.5 and GeoVLA while generalizing to unseen camera views.
ProTakeaways
- VGA replaces VLA language/video backbones with VGGT, a pretrained 3D geometric transformer.
- Reaches 98.1% average on LIBERO, beating π0.5, OpenVLA-OFT, GeoVLA, and Motus.
- Beats OpenVLA-OFT by 26.3 points on LIBERO-Plus unseen-camera tests.
- Real-world: 58% success under unseen viewpoints vs 52% for π0.5, 3% for OpenVLA.
- Uses LoRA rank 64, 500M trainable params, ~100ms latency (10 Hz control).
- Weakness: no LLM backbone, so struggles with commonsense-heavy language instructions. Project page.
Most modern robot policies borrow their brain from somewhere else. Vision-Language-Action (VLA) models bolt a robot head onto a pretrained vision-language model. Video-action models bolt one onto a video diffusion transformer. Both approaches inherit representations optimized for 2D pixels or text tokens, then ask them to output 3D motor commands. A new paper argues this is the wrong starting point entirely, and shows what happens when you swap the backbone for a native 3D world model instead.
The system, called Vision-Geometry-Action (VGA), replaces the language or video backbone with VGGT, a feed-forward 3D geometric foundation model pretrained on multi-view datasets. VGA takes multi-view observations, produces native spatial representations, and conditions action prediction entirely on those representations, establishing a direct vision-to-geometry mapping that translates visual inputs into physical actions. It beats top-tier VLA baselines such as π0.5 and OpenVLA-OFT on standard benchmarks, and holds up in real-world viewpoint shifts where those baselines collapse.

Why 2D brains fumble 3D hands
The core critique is architectural. VLA and video-predictive models rely on backbones pretrained on internet-scale image-text or temporal pixel data. Their representations are shaped by semantic concepts or 2D priors, lacking the precise 3D geometric properties required for physical manipulation. Reaching, grasping, and orienting are defined by positions, rotations, and spatial relationships. A representation optimized to predict the next token or the next frame is not naturally organized around those quantities.
Prior fixes have tried to patch this by feeding in point clouds or depth maps, or prepending a 3D encoder to a VLM. The authors call the latter a "3D-2D-3D transformation loop": rich geometry gets projected into a flat latent space, then decoded back into 3D actions. Extra sensors, meanwhile, introduce noise, fusion complexity, and hardware cost. VGA's bet is that if the backbone is already geometric, none of this patching is necessary.