Microsoft Research Finds Hidden Physics Inside Video Diffusion Models
A new paper shows video diffusion models secretly encode physical laws in their transformer layers — despite never being trained to do so
- Hidden physics signal: Video diffusion models encode physical plausibility linearly in their transformer layers, despite never being trained on any physics objective.
- Beats dedicated world models: WAN-1.3B reaches 81.27% accuracy on IntPhys/InfLevel benchmarks, outperforming V-JEPA 2 (71.36%) and VideoMAE-Large.
- Reverse sampling trick: Researchers recover internal states for real videos by integrating the model's velocity field backwards from clean video to noise.
- Middle layers, early causality: Physics is most readable in middle transformer blocks (15–25), but causally injected by early layers.
- Quantitative physics encoded: Linear regressors on activations predict exact initial velocity and position of simulated trajectories with R²≥0.99.
- Code available: github.com/ParsaEsmati/videodiffusionphysics — works with WAN-1.3B, CogVideoX-2B, and LTX-2B.
Video diffusion models like Sora, Veo, and Cosmos are increasingly pitched as world simulators. But a nagging question has followed that ambition: do these models actually understand physics, or are they just very good at copying motion patterns from training data? A new paper from researchers at the University of Bristol, McGill, Mila, and Microsoft Research has a surprising answer. The physics is in there , and it is more accessible than anyone expected.
The paper, "The Invisible Hand of Physics", probes the internal states of three large-scale video diffusion models and finds that physical plausibility is linearly decodable from their transformer layers. That means a simple linear classifier , no fine-tuning, no complex head , can read off whether a video obeys the laws of physics directly from the model's hidden activations. The catch: this signal is completely absent from the model's input. It emerges inside the transformer during the denoising computation itself.
Why this question is hard to answer
Modern video diffusion models operate in the latent space of a variational autoencoder (VAE) , a compression network trained purely to reconstruct video pixels. That VAE has no reason to organize its latent space around physics. And the diffusion model on top of it is trained only to denoise: to gradually transform random noise into a clean video. There is no explicit physics objective anywhere in the pipeline.
The other problem is access. Diffusion models generate videos by starting from noise and integrating forward. They never "see" a real video as an input , they only produce outputs. So you cannot simply feed a real video through the model and inspect what it thinks about it. The internal states the researchers wanted to probe simply do not exist in normal inference.
The prior state of the art for physics understanding in video models came from a different family entirely: self-supervised predictive encoders like V-JEPA (Meta's video joint embedding predictive architecture). V-JEPA, trained to predict future video states in representation space, achieves 98% zero-shot accuracy on IntPhys and 62% on InfLevel , benchmarks designed to test whether a model can distinguish physically plausible from physically impossible events. The hypothesis was that a predictive training objective was the key ingredient for physics understanding to emerge.
The trick: running the model backwards
The key methodological innovation is reverse sampling. Instead of generating from noise, the researchers take a real video, encode it with the VAE to get a clean latent, and then integrate the model's learned velocity field
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
