Skip to main content
Back

Peking University's VADD Cuts Masked Diffusion Image Error in Half at 10 Steps

A latent variable trick lets masked diffusion models model token correlations, cutting FID by half at 10 sampling steps.

  • VADD adds a latent variable to masked diffusion models to capture correlations between tokens unmasked in the same step
  • Trains via a Double Evidence Lower Bound (DELBO) using an auxiliary recognition network, similar to VAEs
  • Cuts CIFAR-10 FID from 334.3 to 170.3 at 10 sampling steps, and 140.3 to 64.6 at 50 steps
  • Beats a 5M-iteration autoregressive Transformer on LM1B perplexity with only 1M iterations of training
  • Sampling cost is identical to standard MDMs; training cost is roughly 1.5x due to the recognition network
  • Code and paper available on GitHub and arXiv

Masked diffusion models (MDMs) have become one of the more interesting alternatives to autoregressive generation because they can unmask many tokens in parallel instead of predicting one at a time. The catch has always been the same: push the number of denoising steps down to make sampling fast, and quality falls off a cliff. A new paper from Peking University and Huawei proposes a fix borrowed from the VAE playbook, and the numbers are hard to ignore.

The method, called Variational Autoencoding Discrete Diffusion (VADD), cuts FID on CIFAR-10 from 334.3 down to 170.3 at 10 sampling steps against a matched MDM baseline, and reaches the same generative perplexity as the baseline on OpenWebText with less than half the compute. Code is available on GitHub.

Why few-step masked diffusion breaks

A masked diffusion model works by progressively masking dimensions of a data point toward an all-masked state in the forward process, and gradually unmasking multiple dimensions simultaneously in the backward process. That parallel unmasking is what makes them attractive next to autoregressive models that produce one token at a time.

The problem lives in how the denoising step is parametrized. Each backward step is typically modeled as a product of independent categorical distributions across dimensions, which fails to capture complex inter-dimensional dependencies commonly present in real-world data. In plain terms: when the model unmasks several tokens at once, it picks each one independently, as if they don't need to agree with each other.

With many denoising steps that's mostly fine because you only reveal a token or two per step. The issue becomes pronounced when using a small number of denoising steps, where many dimensions must be unmasked simultaneously, amplifying the impact of independence assumptions. Prior fixes typically required guidance from a pretrained autoregressive model at inference, which adds compute at exactly the moment you were trying to save it.