Skip to main content
Back

Pacific Northwest Lab's Granite Vision 3.3 Struggles Where Physics Meets Diagrams

A systematic reward ablation on physics problems shows attention-based supervision doubles spatial reasoning accuracy but breaks symbolic math domains.

  • PNNL researchers systematically ablate four GRPO reward types for physics reasoning in a 2B VLM.
  • Benchmarked on PhyX, a 3,000-problem visual physics benchmark where top models trail humans by 29+ points.
  • Novel attention-weight reward uses the model's own image attention, requiring no spatial annotations.
  • Spatial relation accuracy jumps from 0.27 to 0.50 under the attention reward, the largest gain observed.
  • Same reward degrades symbolic domains (Thermodynamics 0.139, Wave/Acoustics 0.164), suggesting capacity conflict at 2B scale.
  • Rubric rewards produce more coherent reasoning chains but trade off raw accuracy; no reward dominates all domains.

Reinforcement learning has become the default recipe for teaching language models to reason, but almost all the recent breakthroughs (DeepSeek-R1 style GRPO training, verifiable rewards, rubric feedback) have been tested on text-only problems. A new study from Pacific Northwest National Laboratory runs the same playbook on a vision-language model tackling physics, and the results are messier than the language-only literature would suggest. State-of-the-art VLMs fall far short of human performance on physics benchmarks, and how reward design shapes their physical reasoning behavior remains poorly understood.

Why physics breaks vision-language models

Physics problems with diagrams are uniquely hostile to current VLMs. Solving one requires reading a schematic, recalling the right physical law, doing symbolic multi-step math, and producing an answer with correct units. The PhyX benchmark was built to test this combination, and the gap it exposes is brutal: GPT-4o, Claude 3.7 Sonnet, and GPT-o4-mini score 32.5%, 42.2%, and 45.8% respectively, trailing human experts by more than 29 points.

PhyX is deliberately hard because de-redundancy in the benchmark forces models to perceive concepts from one modality only. You cannot cheat by reading the answer out of the caption. The dataset covers six domains (Mechanics, Electromagnetism, Thermodynamics, Wave/Acoustics, Optics, and Modern Physics) in both multiple-choice and open-ended formats.

Four rewards, one small model