Google DeepMind's Frozen Forecasting Reveals Which Vision Models Actually Predict the Future
Google DeepMind built a unified benchmark testing whether frozen vision backbones can predict what happens next, revealing video beats images and synthesis rivals masking.
Takeaways
- Google DeepMind proposes a unified framework to evaluate forecasting in frozen vision backbones.
- Trains latent diffusion on frozen features, decodes with lightweight readout heads per task.
- Benchmarks nine models across pixels, depth, point tracks, and bounding boxes.
- Video-pretrained backbones beat image-only ones on forecasting across nearly every task.
- WALT synthesis model matches or exceeds masked-pretraining models at similar parameter counts.
- Vision-language contrastive supervision does not improve forecasting ability in the tested setup.
Perception models are usually judged on what they can identify in a frame that already exists. A self-driving car or a robot planning an action needs something harder: a sense of what is about to happen. A team from Google DeepMind and the Toyota Technological Institute at Chicago has released Frozen Forecasting, a unified evaluation framework that turns any frozen vision backbone into a video forecaster and measures how well it anticipates the future across four abstraction levels.
Forecasting is inherently stochastic, and any model of the future has to grapple with that uncertainty. Existing evaluations for self-supervised video representations largely test perception on past and present frames, which reveals what a model understands but not what it can anticipate. The authors wanted an apples-to-apples way to compare backbones trained under very different regimes on the specific skill of predicting future states.
Diffusion in the frozen latent space
The setup has two stages. A lightweight attention-based readout head is fit to each frozen model for each downstream task, then a diffusion model is trained to forecast future trajectories directly in the space of the frozen video representations. The diffusion module never sees pixels. It only sees the backbone's own latents, which keeps the comparison architecture-agnostic across image encoders, masked video models, and video synthesis models.

Concretely, the diffusion model is conditioned on the latent encodings of 4 past frames after layer normalization, and models a 16-frame clip that includes the 4 past frames and 12 future frames jointly in time. The readout head is trained only on observed frames and stays fixed during forecasting. That decoupling isolates the forecasting capacity of the backbone itself, and the diffusion transformer is deliberately a plain 5-layer vanilla design so a powerful task-specific forecaster cannot mask a weak representation.