Gaussian Mixture Diffusion Models Can Match Any Target Distribution, Math Proves
A new proof shows that diffusion models using Gaussian-mixture reverse steps with ReLU-parameterized weights can approximate any well-behaved conditional distribution arbitrarily well.
Takeaways
- New theoretical paper proves Gaussian-mixture reverse kernels with ReLU logits are universal for conditional diffusion.
- Addresses known expressive bottleneck of single-Gaussian DDPM reverse steps, especially in few-step regimes.
- Proof combines Norets' mixture density theory with quantitative ReLU approximation bounds via path-space KL decomposition.
- Approximation error decays algebraically in mixture resolution; terminal mismatch shrinks exponentially with diffusion horizon T.
- Complements empirical work like Soft Mixture Denoising and Gaussian Mixture Solvers with population-level guarantee.
- Result is expressivity-only: no code, no training analysis, no statistical estimation or optimization guarantees.
Standard diffusion models like DDPM assume that each denoising step can be modeled as a single Gaussian with a learned mean. That assumption is convenient but restrictive, and a new theoretical paper on arXiv argues we now have solid mathematical ground for replacing it with something more expressive: finite Gaussian mixtures whose mixing weights are produced by a ReLU network.
The main result is a universality theorem. It proves that conditional diffusion models whose reverse kernels are finite Gaussian mixtures with ReLU-network logits can approximate suitably regular target distributions arbitrarily well in context-averaged conditional KL divergence, up to an irreducible terminal mismatch that typically vanishes as the diffusion horizon grows. Translated: if you let each reverse step be a mixture of Gaussians rather than one, and use a neural net to choose the mixing weights, you can in principle match any reasonable target conditional distribution.
One Gaussian per step leaves quality on the table
The motivation comes from a well-known limitation of DDPM-style samplers, where each reverse step is typically a single Gaussian with learned mean and sometimes learned variance. Recent work argues that this ansatz is too restrictive, especially in few-step regimes, and proposes richer reverse transitions based on Gaussian mixtures or related soft-mixture denoisers.