Re2Pix Splits Driving Video Prediction in Two, Hitting 14x Faster Results
Re2Pix splits video prediction into semantic forecasting first, then diffusion rendering, cutting training time up to 14x on driving benchmarks.
ProTakeaways
- Re2Pix decomposes video prediction into DINOv2 feature forecasting followed by diffusion-based pixel synthesis conditioned on those forecasts.
- Achieves 7x faster convergence on FID/FVD and 14x on segmentation mIoU versus a standard diffusion baseline.
- Nested dropout on PCA-ordered feature channels forces reliance on robust coarse semantics, avoiding overfitting to fine details.
- Mixed 90/10 ground-truth/predicted feature supervision closes the train-test gap for autoregressive rollouts.
- Competitive with Cosmos-Predict 2 and beats Vista despite using vastly less pretraining data on driving benchmarks.
- Accepted to ECCV 2026; code at github.com/Sta8is/Re2Pix, weights coming.
Predicting what a driving scene will look like a few seconds into the future is one of the harder problems in generative video. A model has to keep track of what objects exist and how they move, while also rendering photorealistic pixels for each frame. A new paper from researchers at Archimedes/Athena RC, valeo.ai, and the National Technical University of Athens argues that doing both jobs inside a single latent diffusion model is exactly the wrong approach.
Their framework, called Re2Pix, splits the task into two stages: first predict the future in the semantic feature space of a frozen vision foundation model, then use a diffusion transformer to render pixels conditioned on those forecasts. The result is up to a 7x speed-up on generation metrics and 14x on segmentation metrics compared to a standard diffusion baseline, with the paper accepted to ECCV 2026.
The entanglement problem
Most modern video predictors are end-to-end diffusion models that operate in the latent space of a VAE. Semantic structure and fine-grained visual details are deeply entangled within the same representation, so the model must simultaneously infer scene dynamics and render photorealistic appearance. This often leads to temporal inconsistencies such as object identity drift, structural degradation, or flickering artifacts.