Skip to main content
Back

SandboxVLM Lifts GPT-4o Spatial Reasoning by 17 Points Without Retraining

A training-free pipeline turns 2D images into abstract 3D bounding-box scenes, giving GPT-5 and GPT-4o a large boost on spatial reasoning benchmarks.

Paper, Full Access
  • SandboxVLM adds 3D reasoning to VLMs by handing them abstract bounding-box renderings of the scene, no training required.
  • Four-stage pipeline: multi-view diffusion, proxy elevation, cross-view voting, and 3D-aware prompting.
  • Boosts GPT-4o by 17.4 points on SAT-Real and beats MindJourney by 8.3 points at the same task.
  • Also outperforms RoboBrain2.0-32B, a model fine-tuned specifically for spatial understanding.
  • Ablations show text coordinates can beat rendered visuals, exposing VLMs' language-centric bias.
  • Limitations: static scenes only, no physical properties, and errors originate mostly from VLM pointing step.

Vision-language models can name every object on your desk, but ask them whether the mug is closer than the monitor or which way you would turn to face the window, and they start guessing. A team from Tsinghua, Harvard, and MIT argues the fix is a lightweight scaffold that hands the model a coarse 3D sketch of the scene and lets it reason on top of that, rather than piling on more 3D training data.

SandboxVLM is a zero-shot, plug-and-play pipeline that wraps abstract 3D bounding boxes around scene elements to encode geometric structure and kinematics for a VLM. No fine-tuning, no new weights, no point-cloud encoder bolted onto the backbone. On the SAT-Real spatial reasoning benchmark, it lifts GPT-4o by 17.4 points over its own baseline and beats the previous test-time scaling method by 8.3 points.

Why VLMs keep tripping over 3D

The authors frame the problem as a modality gap. Models like GPT-5 and Qwen3-VL are trained almost entirely on 2D images and text, so they interpret the world as projections rather than volumes. That breaks down on anything requiring reasoning under a viewpoint change, estimating relative position, or predicting what happens if an object moves.

Prior fixes have taken two routes. One camp injects point clouds or multi-view features directly into open-source VLM backbones, which requires dense 3D supervision and locks you out of proprietary models like GPT-5. The other camp, exemplified by MindJourney, uses video diffusion to imagine extra 2D viewpoints. Useful, but the model still reasons over 2D frames.

Coarse structure is enough

The paper's guiding intuition is that humans do not build metric 3D models to catch a ball or find a chair. We use coarse relational understanding. SandboxVLM applies that principle by representing each scene with a compact set of abstract 3D bounding boxes that encode spatial arrangement and physical dynamics while discarding low-level visual details. The design retains only the structural cues needed for reasoning and leaves appearance to the original image.