Skip to main content
Back

NVIDIA's SpatialClaw Beats Every Spatial AI Agent by 11 Points Without Training

NVIDIA Research's SpatialClaw uses a persistent Python kernel as its action interface, beating prior spatial agents by 11.2 points across 20 benchmarks with no training required.

Repo, Full Access
  • NVIDIA Research releases SpatialClaw, a training-free agent that uses a live Python kernel as its action interface for spatial reasoning tasks.
  • +11.2 point gain over the prior best spatial agent across 20 benchmarks, with no benchmark- or model-specific tuning.
  • Persistent kernel design lets the agent compose perception outputs as variables, inspect intermediate results, and revise its strategy mid-run -- unlike fixed tool-call APIs.
  • Works across six VLM backbones (Qwen3.5/3.6, Gemma4, 26B to 397B parameters) with consistent gains on 19 of 20 benchmarks.
  • Open-source code on GitHub includes full agent runtime, all 20 benchmark loaders, and GPU perception server; non-commercial license.
  • Key finding: over 70% of wins over structured tool-call trace to code composition and control flow -- capabilities a fixed JSON API cannot express.

Spatial reasoning -- figuring out where things are, how they relate in 3D space, and how they move -- is one of the hardest unsolved problems for vision-language models (VLMs). Most agents that try to tackle it hit the same wall: they call a fixed menu of tools in a rigid order, and the moment their initial plan is wrong, there's no way to recover. SpatialClaw, a new training-free framework from NVIDIA Research, proposes a different answer: give the agent a live Python kernel and let it write code.

The bottleneck was never the tools

Spatial reasoning remains a fundamental challenge for VLMs, and tool-augmented agents attempt to address this by augmenting them with specialist perception modules -- yet their effectiveness is bounded by the action interface through which those tools are invoked. SpatialClaw's core argument is that the interface itself, not the tools, is what limits performance.

To understand why, consider the two dominant approaches before SpatialClaw:

  • Single-pass code: The agent writes a complete Python program in one shot, before seeing any intermediate results. If the initial assumption is wrong, the error propagates all the way to the final answer.
  • Structured tool-call: The agent dispatches tools through a fixed JSON API. It can call tools, but it can't freely chain their outputs, apply NumPy operations on the results, or branch based on what it sees.

SpatialClaw treats code as the action interface: a VLM-backed agent writes one Python cell per step into a persistent Jupyter kernel pre-loaded with perception primitives (SAM3 segmentation, Depth-Anything-3 reconstruction, geometry utilities) and scientific libraries (NumPy, SciPy, Matplotlib). Each cell can compose tool outputs, inspect intermediate evidence, and revise the analysis before committing an answer with ReturnAnswer(...).

Win/loss chart comparing SpatialClaw vs single-pass code and structured tool-call across 14 spatial reasoning task categories

A five-stage loop, not a one-shot guess

The agent's runtime is built around a five-stage agentic loop. A planner drafts a strategy, the main VLM writes one Python cell, that cell is AST-checked for safety and executed in the stateful kernel, and the outputs -- stdout, new variables, and any show() images -- flow back as the next observation. This repeats until the agent calls ReturnAnswer(...).

The persistent kernel is the key architectural detail. Variables computed in step 1 are still in scope in step 5. If the agent segments an object in one cell and reconstructs depth in the next, it can combine both results with a single NumPy expression -- no re-running, no re-querying. Real agent sessions on the project page show trajectories with over 100 cross-step variable references and 18 intermediate show() calls on a single question.

At deployment scale, three independent services coordinate: a vLLM backbone serving the VLM, a GPU perception-tool server running SAM3 and Depth-Anything-3, and the agent runtime managing the Jupyter kernels. All three can run on a single GPU machine or across a SLURM cluster.

+11.2 points, no tuning, six backbones

Across 20 spatial reasoning benchmarks, SpatialClaw reaches 59.9% average accuracy, outperforming the prior best spatial agent by +11.2 points -- with the same system prompt, tool set, and hyperparameters across all benchmarks and six VLM backbones. Those backbones span Qwen3.5 and Gemma4 models ranging from 26B to 397B parameters.

The gains aren't uniform -- they concentrate exactly where you'd expect a compositional interface to help most:

  • DSI-Bench (4D reasoning): +17.6 pp
  • MindCube (multi-view): +15.3 pp
  • MMSI (multi-image spatial): +13.4 pp

The ablation table is particularly telling. Using the same toolset and the same prompt, only changing the action interface:

Action InterfaceAvg. AccuracyDelta
No-tool baseline53.4%--
Single-pass code55.2%+1.8
Structured tool-call56.7%+3.3
SpatialClaw (code as action)59.9%+6.5

Why does code win? The attribution breakdown

The team used an LLM judge (Gemini 3 Pro) to analyze every case where SpatialClaw was correct but structured tool-call failed, and assigned each win to a root cause. The breakdown:

  • Code composition (52.2%): Chaining multiple tool calls into a single coherent program -- something a fixed JSON API simply can't express.
  • Interface-neutral (28.3%): Cases where either interface could theoretically solve the problem, but SpatialClaw's flexibility still helped.
  • Control flow (19.5%): Using if/for branching over intermediate results to adapt the analysis mid-run.
Horizontal bar chart showing SpatialClaw win attribution: Composition 52.2%, Interface-neutral 28.3%, Control flow 19.5%

Over 70% of wins trace directly to capabilities a fixed-API tool interface cannot provide. Notably, removing all utility wrapper functions (like tools.Mask or tools.Geometry) and keeping only raw NumPy/SciPy barely changes the score -- the interface design matters far more than the specific helper functions bundled with it.

Spontaneous task adaptation, for free

One of the more surprising findings: the agent spontaneously adapts which mathematical primitives it reaches for, purely based on the question semantics, with no category-specific prompting or tool routing. Distance questions invoke KDTree search and vector norms. Direction questions rely on dot products and angular operations. Camera motion questions draw on pose composition and transformation chains. This kind of task-adaptive behavior is exactly what rigid tool-call interfaces struggle to elicit.

Heatmap of NumPy/SciPy function usage frequency across computer vision task categories, showing spontaneous task-adaptive tool selection

Getting started

SpatialClaw is a training-free spatial reasoning framework available as open source on GitHub. The official repo ships the full agent runtime, all 20 benchmark loaders, perception tool wrappers, a FastAPI-served GPU tool server, and vLLM auto-discovery. Setup takes 15-30 minutes:

# 1. Clone with submodules and install
git clone --recursive https://github.com/NVlabs/SpatialClaw.git
cd SpatialClaw
bash spatial_agent/scripts/setup.sh
# 2. Add API keys (or use self-hosted vLLM)
cp .env.example .env
# 3. Run an experiment
python -m spatial_agent.entrypoints.run \
    --dataset spatial_agent/config/dataset/erqa.json \
    --model   spatial_agent/config/model/gemini-3-pro.json \
    --concurrency 4

The code is released under NVIDIA's Source Code License (non-commercial). It supports any VLM accessible via vLLM or an API key, so you can swap in Qwen, Gemma, or any compatible model without touching the agent code. The perception backend requires a GPU for running SAM3 and Depth-Anything-3.

What this means for the field

SpatialClaw's result carries a broader implication: the community has been benchmarking spatial agents while conflating two separate variables -- the quality of perception tools and the expressiveness of the action interface. These results suggest the interface may be the dominant factor. An agent with mediocre tools but a flexible code interface outperforms one with excellent tools locked behind a rigid JSON API.

The practical use cases are concrete: robotic scene understanding, autonomous navigation, video analysis for surveillance or sports, multi-view 3D reconstruction pipelines, and any application where an agent needs to reason about where things are relative to each other across frames or viewpoints. The training-free design means you can drop it on top of whatever VLM you're already using without any fine-tuning budget.