InPhyRe Exposes How Frontier AI Ignores Video and Fakes Physics
A new benchmark tests whether vision-language models can infer novel physics from demonstrations, and finds nearly all of them lean on text shortcuts instead of watching the video.
ProTakeaways
- InPhyRe is the first VQA benchmark measuring whether LMMs can infer novel physics from a few demonstration videos.
- The benchmark uses PyBullet and Blender to generate collisions that violate momentum conservation and other real-world laws.
- 13 open-source LMMs plus GPT-4.1 and Gemini 2.5 variants all fail on scenarios that break their training-time physics priors.
- Replacing demonstration videos with random unrelated frames changes accuracy by less than 1 point in 86.8% of cases.
- Gemma3-12B pays roughly 10 times less attention to image tokens than text tokens, explaining the language bias.
- Human subjects scored above 90% with just one visual demonstration and no text, exposing a large model-human gap.
Ask a large multimodal model what happens when two billiard balls collide head-on and it will probably give the right answer, thanks to momentum conservation being baked into its training data. What happens on a snowy road, in low gravity, or in any environment where the physics differs from what the model has seen? A new benchmark called InPhyRe tries to answer that question, and the results are unflattering for today's frontier vision-language systems.
InPhyRe is the first visual question answering benchmark that evaluates how well LMMs can infer the underlying physics from demonstration samples and use it to make physical reasoning predictions. The authors evaluated 13 open-source models plus GPT-4.1 and Gemini 2.5 variants, and almost all of them fail to adapt when the physical laws in the test scene differ from the ones they memorized during training.
Knowing physics vs. applying it
The paper draws a sharp line between two capabilities. Parametric knowledge is the physics a model absorbs during training, such as conservation of momentum. Inductive physical reasoning is the ability to look at a handful of example videos, work out the rules governing that particular world, and apply them, even when those rules contradict the model's prior. Humans do this constantly. Drivers adjust braking distance for wet roads. Toddlers recalibrate force when a drawer sticks. LMMs, it turns out, mostly do not.

The distinction matters because existing physics benchmarks quietly conflate the two. Most rely only on true physical laws that are possibly encoded in the parametric knowledge of LMMs, which confounds any evaluation of inductive reasoning. If a model gets a CLEVRER-style collision question right, you cannot tell whether it actually reasoned about the video or just recited memorized physics.