Sun Yat-sen's SIRI-Bench Exposes How Badly VLMs Fail at 3D Geometry
A new 9,000-question benchmark hides geometry conditions inside 3D scenes, exposing that top vision-language models fail more than half the problems.
PRO- SIRI-Bench introduces 9,078 video-question-answer triplets built from solid geometry problems rendered as 3D scenes. Paper
- The best VLM tested, Doubao-1.6-Vision, only lands 31% of predictions within 40% error; over 33% exceed 100% error.
- Sibling text-only LLMs solve up to 89% of problems within 20% error, isolating spatial perception as the bottleneck.
- An Automatic Scene Creation Engine chains Qwen-Math and Qwen-LM agents to generate Blender scripts and render scenes.
- GPT-4o misidentifies the geometric shape in roughly 79% of problems and invents unsupported constraints during reasoning.
- All models score under 20% on volume questions because dimension errors get cubed in the calculation.
Vision-language models have been racking up impressive scores on math and reasoning benchmarks, but those tests almost always hand the model every fact it needs in clean text. A new benchmark from Sun Yat-sen University buries the important conditions inside a rendered 3D scene, and the results are ugly: even OpenAI o3 and Doubao-1.6-Vision get more than half the questions wrong by a wide margin.
The benchmark, called SIRI-Bench (Spatial Intelligence ReasonIng Benchmark), takes a simple but pointed approach. Instead of describing a solid geometry problem in words or drawing a 2D diagram, the authors render the actual 3D object as an 8-second video and strip the numerical dimensions out of the question text. To answer, a model has to look at the scene, figure out what shape it is, estimate its measurements, and then do the multi-step math on top of that.
Why textual math benchmarks were letting VLMs off easy
Spatial intelligence refers to the capacity to interpret spatial information, encompassing skills such as perceiving shapes, understanding transformations, and applying spatial knowledge. Prior spatial benchmarks mostly probed what the authors call intuitive intelligence, things like distance estimation or layout understanding, without forcing models to chain that perception into multi-step reasoning. Math-heavy benchmarks like MathVista and MATH-Vision target symbolic and geometric reasoning through plane geometry, but remain abstract and detached from real-world contexts.
SIRI-Bench sits in the gap. Each of the 9,078 video-question-answer triplets is a solid geometry problem where the geometric conditions have to be inferred from a rendered video rather than read from the prompt.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
