CausalPhys Exposes Why Top Vision AI Models Fail at Physical Reasoning
CausalPhys introduces 3,000+ expert-annotated questions and a new fine-tuning method that teaches VLMs to reason about why things happen, not just what happens next
- New benchmark: CausalPhys introduces 3,000+ video/image questions across 4 causal domains, each paired with expert-annotated causal graphs.
- Beyond accuracy: A new 4-metric evaluation (ACC, EF, RA, DC) measures whether a model's reasoning chain actually follows correct causal dependencies.
- Key finding: All tested VLMs -- including GPT-4o and Gemini-2.5-Flash -- show high entity recognition (EF ~0.7) but very low causal relation awareness (RA ~0.2-0.3).
- Open = closed: Open-source models like InternVL3 match proprietary systems on causal reasoning; scale alone doesn't solve the problem.
- CRFT fix: Causal Rationale-informed Fine-Tuning improves Qwen2-VL 7B accuracy from 53% to 71% while preserving interpretability metrics that standard SFT destroys.
- Open release: Benchmark data, annotation pipeline, causal graph schemas, and evaluation scripts are all publicly available.
Ask a state-of-the-art vision-language model whether two cars are about to collide, and it will give you a confident, well-written answer. It will often be wrong. The model has learned to sound right, but it hasn't learned to reason about causes and effects the way a human does. A new paper tackles this gap head-on with CausalPhys, a benchmark and training framework designed to measure and fix how VLMs understand the physical world at a causal level.
Why getting the answer right isn't enough
Current limitations in VLM causal reasoning stem primarily from insufficient structural guidance rather than a lack of reasoning capacity. The problem isn't that these models are dumb -- it's that they've been trained to chase statistical patterns in data rather than to model the underlying mechanisms that make things happen. Reasoning about how the world changes under interventions or viewpoint shifts is naturally framed through causal inference, yet existing benchmarks rarely include ground-truth causal annotations, making it impossible to rigorously measure whether models follow correct causal dependencies.
This matters enormously for anyone building systems that interact with the physical world. A robot that can describe a scene but can't reason about what happens if it pushes an object is fundamentally unreliable. CausalPhys is the first benchmark to couple real-world physical reasoning questions with explicit ground-truth causal graphs, enabling interpretable, mechanism-level evaluation beyond surface-level answer accuracy.
Pearl's ladder, applied to video
The benchmark's structure is grounded in Judea Pearl's "causal ladder" -- a framework distinguishing three levels of reasoning: observing (what is), intervening (what if I do X), and counterfactual (what if things had been different). CausalPhys spans four complementary categories: Perception, Anticipation, Intervention, and Goal Orientation, across 16 subfields. Each of these maps to a different rung of that ladder.
- Perception (951 questions): "What is..." -- identifying observable states, optics, containability, scene reconstruction, mechanics
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
