Odyssey's CaliBench Catches Top Video AI Models Failing as Stochastic Simulators

A dice roll looks random, but the distribution of outcomes is not. Roll it thousands of times and each face shows up roughly one-sixth of the time. If a video world model is meant to stand in for reality, it should reproduce that spread. CaliBench, a new benchmark from Odyssey now peer-reviewed and published in Transactions on Machine Learning Research, tests exactly this and finds that today's best models are failing badly.

The core claim is uncomfortable for anyone building simulators on top of generative video: the paper shows that across six state-of-the-art models tested on nine classic stochastic environments, current architectures routinely fail as reliable stochastic simulators, and even when individual trajectories look highly realistic, they systematically suffer from severe probability mass over-concentration and mode collapse.

Why per-frame realism was hiding the problem

Existing video benchmarks tend to do one of two things. They score individual clips for sharpness, temporal consistency, and physical plausibility, or they compare whole datasets in a learned feature space like Fréchet Video Distance. Both leave the fine-grained aleatoric uncertainty of specific phenomena untested, where aleatoric uncertainty is the irreducible randomness in a physical process itself, as opposed to uncertainty from not knowing the model's parameters.

The gap matters because a model can ace every per-sample check and still be badly miscalibrated. Imagine a generator that produces gorgeous dice-roll videos but always lands on a four. Every frame passes review. The distribution is broken. That is the failure mode CaliBench is designed to surface.

Physics with a known answer key

The trick is to pick physical systems where the true outcome distribution is known analytically, then check how far a model's samples sit from it. The team curated outcome spaces with closed-form references: binomial Galton boards, Bernoulli forks, uniform dice, cards and lottery draws, and a skewed European-roulette colour distribution, giving an exact calibration test rather than a fuzzy feature-space comparison.

The evaluation pipeline is straightforward:

1. Set up a scene with a known reference distribution (e.g. a fair six-sided die).
2. Generate 32 videos per scene per model from the same conditioning frame.
3. Extract the discrete outcome from each video using Gemini 3.1 Pro as a vision-language reader, validated against human annotators at roughly 93% agreement.
4. Score the empirical distribution against the analytical reference using total variation distance, plus a significance test to rule out sampling noise.

Two failure modes, cleanly separated

One of the paper's most useful contributions is refusing to blur two very different things into one number. CaliBench reports:

• Scorability: the fraction of generations that produce a clean, readable outcome at all. A pendulum that tears itself apart is not scorable.
• Calibration: among readable outcomes, how far the empirical distribution sits from the reference, measured by total variation distance.

These get folded into a single headline metric called mean normalised total variation (mnTV), so models can be ranked across all nine scenes at once. A perfectly calibrated model would score zero.

What the frontier looks like

The lineup includes WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, and Cosmos3-Super, tested on all nine scenes with 32 generations each. The results are blunt. Most scene-model pairs are significantly miscalibrated. On Galton boards, models over-concentrate on the central bin well past its true 24.6% probability. In the most extreme case, Veo 3.1 exhibits total mode collapse on the die scene, outputting the exact same face on every seed.

There is no clean winner. SeeDance performs best in aggregate, but HappyHorse wins some individual scenes, and the team found no simple rule predicting which model would handle which physics. One knob did behave predictably: classifier-free guidance, the technique that pushes diffusion outputs toward the conditioning signal. Turning it down reduces over-concentration and increases entropy, but the videos become structurally messier, so scorability drops. Realism and calibration currently trade off against each other.

Why this matters for anyone using world models as simulators

If world models are going to stress-test self-driving stacks or robot policies in simulation, the whole point is to surface rare, dangerous tail events. A simulator that collapses onto the single likeliest future hides exactly those. You get a world that looks safer than reality, which is arguably worse than no simulator at all.

What CaliBench is not: it evaluates marginal outcome distributions on curated toy-physics scenes, not full continuous trajectory spaces or open-world driving scenes. It measures marginal outcome distributions rather than full continuous trajectory spaces, so a model could pass CaliBench and still be miscalibrated on more complex joint distributions. It is also compute-hungry: the current matrix required roughly 5,184 VLM queries across models and seeds.

Getting the code

The full protocol is now on GitHub, including the curated conditioning frames, the standardised VLM extraction prompts, analysis code, and the mnTV metric. The intent is that new models can report mnTV alongside FVD and VBench and be compared directly to the six baselines in the paper. If you are training a video generator and you care about it working as a simulator rather than a highlight reel, this is a cheap way to find out how calibrated it actually is.