Apple and EPFL's VideoFlexTok Generates 10-Second Videos With 8x Fewer Tokens
Apple and EPFL researchers propose a video tokenizer that packs 10-second clips into 672 tokens, cutting training compute by up to 10x.
Takeaways
- VideoFlexTok tokenizes videos into a variable-length, coarse-to-fine sequence instead of a fixed 3D grid.
- First tokens emergently capture semantics and motion; later tokens add fine detail.
- Matches larger models' quality with a 5x smaller (1.1B vs 5.2B) text-to-video AR model.
- Encodes 10-second, 81-frame videos in only 672 tokens, 8x fewer than 3D grid baselines.
- Built with time-causal Transformer encoder, nested dropout, FSQ, and a rectified-flow decoder with REPA (DINOv2 alignment).
- Paper and site available at videoflextok.epfl.ch; code not yet released.
Video generation models spend most of their compute learning to predict pixel-level details that nobody notices. A new tokenizer from Apple and EPFL called VideoFlexTok sidesteps this by representing videos as a variable-length, coarse-to-fine sequence where the first few tokens carry semantic and motion information, and later tokens fill in the fine details. The result: comparable generation quality with a model 5x smaller, or 10-second videos generated with 8x fewer tokens than a standard 3D grid tokenizer.
The pixel-by-pixel tax
The de facto approach to video tokenization represents a video as a fixed-size spatiotemporal 3D grid, each token corresponding to some local information in the original signal. Every video, whether a static shot of a coffee cup or a chaotic action scene, gets the same budget. Most tokenizers also aim for accurate reconstruction, prioritizing pixel-level detail. A downstream text-to-video model that consumes those tokens has to simultaneously predict low-level texture and the more abstract structure like overall semantics and motion.
That mismatch is why frontier video models are so expensive to train. The generator wastes capacity predicting texture noise it could have gotten from a decoder for free.
Tokens that grow up in order
VideoFlexTok's trick is a coarse-to-fine hierarchy. The first tokens emergently encode the most salient semantic, geometric, and motion information, while later tokens add fine-grained details. Reconstruct from 1 token per latent frame and you get the correct scene layout and camera motion. Add more tokens and colors, textures, and finer object details progressively snap in.

To confirm this isn't hand-waving, the authors run a nice probing experiment. They keep only 1 or 2 tokens from a source video, then edit its first frame (turning an orange into an apple with