NVIDIA’s Physis-Lang teaches video models through physics-aware captions

NVIDIA researchers introduced Physis-Lang, an open framework that encodes physical explanations in the captions and prompts used by video-generation models. The work targets a persistent weakness in generated video: realistic frames can still show butter melting upward, blocks passing through one another, or cloth moving without plausible weight and tension.

Training captions provide supervision about objects, actions, and temporal changes. Physis-Lang enriches that supervision with cause, governing principle, and effect. A melting-butter caption, for example, describes heat raising the temperature, the material changing phase, gravity deforming the softened mass, and a liquid pool forming. The framework uses this language to recaption training videos and expand user prompts before generation.

Better prompts lift physics scores

In the reported Physics-IQ Verified snapshot, Physis-Lang configurations of Cosmos 3 Super and Cosmos 3 Nano occupy the top two positions. The benchmark evaluates how faithfully generated videos follow real-world physical behavior.

An inference-only ablation isolates the effect of prompt expansion. Adding physics-aware reasoning to the input increased Cosmos 3’s PhyGenBench score by 5.62 points with fixed model weights and no additional training. This path can be applied to an existing checkpoint because it changes the conditioning text sent to the generator.

A critic rewrites the captioning rules

The framework shares one physical-language representation across three connected stages, allowing critic feedback and failure-driven retrieval to improve the text used for training and generation.

1. Refine the language. A physics-aware critic checks each caption for missing claims and statements unsupported by the video. An agent updates the captioning guidelines, while the captioner remains frozen during this loop.
2. Retrieve weak phenomena. Failed generations become search queries. Language-guided retrieval selects visually diverse videos covering the physical categories where the system performs poorly.
3. Train and generate. The revised guidelines recaption training data and expand inference prompts. Scene-specific negative descriptions identify implausible dynamics for the generator to suppress.

PhysCapBench measures whether the captions improve independently of video generation. The benchmark contains 246 reviewed videos and 3,794 human-verified physical assertions, averaging 15.4 assertions per video. Evaluators split each generated caption into atomic claims, check those claims against the source video, and calculate precision, recall, and F1, which balances the two.

Failures choose the next training clips

On VideoPhy-2 Hard, targeted retrieval raised joint scores in all eight reported categories. Gains ranged from 1.23 points for elasticity to 8.00 points for thermal and chemical processes.

The largest gains occur in processes governed by partially hidden state variables such as temperature, material phase, and chemical composition. Explicit causal descriptions provide conditioning signals for transitions that individual frames may not reveal directly.

Gains carry across backbones

Table 5 of the release paper reports positive mean gains for every tested backbone, spanning models from 4 billion to 64 billion parameters.

Physis-Lang with Cosmos 3 Nano exceeds Veo 3.1 on PhyGenBench but falls slightly short on VideoPhy-2 All.

The benchmarks set clear boundaries

The 5.62-point ablation measures prompt expansion with fixed weights. The full pipeline also uses recaptioned data and failure-driven retrieval, so its results reflect the combined effect of language refinement, data selection, training, and inference-time prompting.

The quantitative evidence covers selected benchmark phenomena, and the top Physics-IQ Verified score remains 48.2%. General simulation ability, long-horizon consistency, and performance on unseen physical processes remain open evaluation questions. Held-out phenomena and human review would help determine how well the gains transfer beyond the reported test sets.

Captions become a control surface

Physics-aware captions give developers a model-agnostic way to supply causal structure. A caption that names forces, state changes, and expected outcomes provides more temporal guidance than a surface description of visible objects and actions. The approach operates at the data and prompt layers, allowing it to sit on top of existing checkpoints.

An implementation can begin with inference-time prompt expansion: define a domain-specific causal schema, expand incoming prompts, keep the generator fixed, and evaluate on a held-out suite. Training integration adds critic-guided recaptioning, retrieval of failure cases, and another model-training run. The same workflow can be tested in other conditional generative systems that use scene descriptions.

The project page and paper provide the framework details, benchmark definitions, ablations, and generated-video galleries covering collisions, tearing, wetting, and deformation under load.