NVIDIA's Spatial-IQ Exposes Why Top AI Models Score 17.7% Where Humans Hit 82%

Ask a person to count every box in a stack of crates, including the ones hidden underneath, and they will score around 82%. Ask the best multimodal AI model available today, and it will score 17.7%. That gap is the central finding behind Spatial-IQ, a new diagnostic benchmark from NVIDIA Research that, unlike most AI benchmarks, opens the black box and shows you exactly where a model's reasoning breaks down.

Why a single score hides the real problem

Most spatial reasoning benchmarks hand a model an image and ask for a final answer. If the model gets it wrong, you have no idea why. Was it a perception failure, where the model could not tell where one object ended and another began? Or was it a reasoning failure, where the model could not infer that a hidden block must exist to physically support the ones above it? These are very different problems, and a single accuracy score cannot distinguish them. Spatial-IQ was built specifically to answer that question.

Nine sub-tasks, one target

The benchmark is grounded in developmental psychology. Humans counting a stack of boxes decompose the problem: identify clusters, parse columns and layers, count what is visible, then infer what must be hidden underneath for the structure to be physically stable. Spatial-IQ mirrors that decomposition, breaking 3D object counting into nine perceptual and cognitive sub-tasks organized by the developmental stages of human spatial cognition. NVIDIA Isaac Sim procedurally generates a diverse dataset of roughly 80,000 stacked 3D structures with per-task ground truth labels.

The nine sub-tasks form a clear hierarchy:

• S1–S4 (Perceptual grouping): Object individuation, clustering, counting columns, counting layers
• S5 (Visible count): How many objects can the camera actually see?
• S6–S8 (Structural reasoning): Which objects are in the top layer, which directly support them, which form load-bearing columns?
• S9 (Hidden count): How many objects are occluded but logically required for physical support?
• T1 (Object count): The final total, visible plus hidden
• T2 (Mental rotation): A separate probe for viewpoint transformation

Objects are constrained to a 4x4x4 voxel grid. A pixel-level depth-buffer occlusion test identifies hidden objects, retaining any object that is either camera-visible or required to support a retained object above it. This guarantees every hidden object in the scene is logically demanded by physical support, not just statistically likely. The benchmark uses a 3,000-sample evaluation set, with 68,000 samples reserved for training.

Models cheat their way to the right answer

When NVIDIA evaluated eight frontier models, including Gemini 3 Pro, GPT 5.4, Claude Opus 4.6, and Qwen3.5-27B, the results revealed something more troubling than low scores. Models were reaching their scores through completely different mechanisms than humans use.

Qwen and Gemini post comparable Object Counting accuracy (17.7% vs 14.8%), yet their Summation Mechanism exact-match rates differ by more than 3x (64.2% vs 18.6%). Qwen is doing something close to correct compositional reasoning. Gemini is hitting a similar headline number through a shortcut. A single accuracy figure would never surface this distinction.

The multiple-choice format exposed an even starker failure. Under five-choice MCQ, every model collapses to chance on Object Counting while humans hold 86.2% accuracy. Models also showed systematic wrong-answer preferences, consistently merging two columns into one or hallucinating a phantom block above the top layer. These are specific perceptual failure modes, not random noise.

Using the hierarchy as a training signal

The most practically useful part of Spatial-IQ is what happens when you train against the sub-task hierarchy rather than the final answer alone. The team fine-tuned Qwen2.5-VL at both 7B and 32B scale under three conditions:

1. SFT-plain: Supervised fine-tuning on the final integer count only, with no decomposition
2. SFT-CoT: Fine-tuning with a chain-of-thought target that walks through each sub-task in order
3. DAPO-tight: Brief SFT warmup followed by reinforcement learning with verifiable rewards (GRPO under the DAPO recipe)

SFT-plain reaches 40.1% on Object Counting at 32B but collapses to 0% on the Internal Referential Chain, Hidden Support Hierarchy, and Summation Mechanism sub-tasks. It found a route to the right total that bypasses the decomposition entirely, a shortcut rather than a skill.

DAPO-tight at 32B reaches 62.6% Object Counting accuracy, up from a 2.9% zero-shot baseline, with the Internal Referential Chain at 95.0% and Summation Mechanism at 93.3%. The reward signal only checks the final integer, yet the RL process naturally discovered that walking through the sub-task chain was the most reliable path to a correct answer.

What transfers beyond this benchmark

Spatial-IQ is scoped specifically to stacked 3D object counting. The authors are explicit that the hierarchy faithfully decomposes Object Counting but not Mental Rotation, suggesting that compositional spatial tasks and transformational spatial tasks recruit separable competencies. A different hierarchy would be needed for navigation, depth estimation, or scene layout tasks.

The underlying workflow, however, transfers directly to other domains:

• Pick a target spatial task
• Decompose it into its prerequisite sub-capabilities
• Use that hierarchy as both a diagnostic and a training signal
• Verify that improvements reflect genuine composition, not a better shortcut

There is also an interesting result buried in the data. VLA-0 outperforms its Qwen3B backbone across the support-relation sub-tasks despite being trained only on robot trajectories. This hints that action-grounded training transfers positively to spatial reasoning, and that embodied training data of the kind used in robotics may be an effective path toward better spatial understanding in vision-language models.

How to get started

Everything is open. The paper is on arXiv, the dataset is on Hugging Face, and the code is on GitHub. The 68,000-sample training split is available separately from the 3,000-sample evaluation set, so you can fine-tune your own models without contaminating the benchmark.

For teams building multimodal systems that need to reason about physical space, whether in robotics, AR/VR, warehouse automation, or medical imaging, Spatial-IQ offers a concrete way to stop guessing why your model fails and start fixing the right thing.