Skip to main content
Back

ProPhy Fixes AI Video Physics, Boosting Dynamic Accuracy by 19.7%

A new framework injects fine-grained physics awareness into video diffusion models, lifting physical commonsense scores by nearly 20% on VideoPhy2.

Pro

  • ProPhy adds a physics branch to video diffusion models with two-stage Mixture-of-Physics-Experts routing.
  • Boosts VideoPhy2 Joint score by 19.7% on CogVideoX-5B; Dynamic Degree jumps from 46.8 to 72.0.
  • Uses Qwen2.5-VL-32B attention maps as supervision for token-level physical expert routing.
  • Semantic experts operate at video level; Refinement experts operate per-token for localized physics.
  • Code released at github.com/zijunwa/ProPhy, works with Wan2.1 and CogVideoX backbones.
  • Expert inversion experiments show distinct experts encode distinct physics like rigidity or cloth deformation.

Video generation models can render a photorealistic frame, but ask them to show a ball bouncing on sand or coffee pouring near a campfire and the physics fall apart. A team from Sun Yat-sen University, Peng Cheng Laboratory, MBZUAI, ETH Zürich, and Lenovo Research has released ProPhy, a training framework that bolts a physics reasoning branch onto existing video diffusion backbones and delivers double-digit gains on physics-focused benchmarks.

The core insight is that current models respond isotropically to physical prompts and neglect fine-grained alignment between generated content and localized physical cues. When a prompt describes several physical phenomena happening in different parts of a scene, existing models blur those cues into a single global signal instead of routing them to the specific regions where they occur.

Where prior physics-aware video models fall short

Recent efforts have taken a few different approaches. VideoREPA distills physical knowledge from foundation models, PhysMaster and PISA use reinforcement learning and reward modeling to induce physical behaviors, and PhysHPO applies a hierarchical preference alignment framework. These bake physics into the weights during training but provide no explicit physical guidance at inference time, which shows up as broken dynamics in complex scenes.

WISA introduces explicit guidance by identifying physical categories from prompts and using a Mixture-of-Physics-Experts (MoPE) structure, but it operates at the whole-video level. It struggles to capture fine-grained processes when multiple physical phenomena coexist or occur within localized regions. Consider a discus throw where dust should only puff up on impact, not trail the disc through the air.