Kyutai's Structure-Agnostic Distillation Beats Dense Matching for Diffusion Image Models

An autoencoder shapes the latent space in which a diffusion model learns to generate images. When that compressed representation is poorly organized, the diffusion model spends more capacity learning its structure. Kyutai researchers have proposed a simpler distillation method that produces diffusion-friendly latents without forcing the autoencoder to copy a vision teacher’s patch layout.

The new paper, accepted at the NeurIPS 2026 Workshop on Principles of Generative Modeling, replaces position-by-position feature matching with image-level alignment. Its pooled objectives match one descriptor per image, allowing the student latent and teacher representation to use different dimensions, token counts, and layouts.

Why patch grids get in the way

The VF loss used by VA-VAE distills a frozen vision encoder such as DINOv2 into an autoencoder. Each position in the student latent must match the teacher feature at the corresponding position. This gives the diffusion model a more structured latent, but it assumes both representations share the same two-dimensional patch grid.

That requirement complicates 1D sequence latents, video representations with temporal axes, and teachers from other modalities. SoftVQ-VAE handles mismatched structures by replicating latent tokens and learning a projection onto the teacher’s grid, adding an artificial correspondence that the underlying representations do not naturally provide.