Waseda Researchers Explain Why Stable Diffusion Forgets Using Hopfield Energy
A new theoretical framework uses Hopfield energy to predict which samples diffusion models forget first — and which to replay to prevent it
Takeaways
- Researchers prove that high-energy, outlier-like samples in diffusion models are forgotten more than typical cluster samples after task changes.
- The framework uses modern Hopfield network energy as a tractable, model-native proxy for forgetting in diffusion models.
- A new concept, "intrinsic forgetting," is defined as the increase in Hopfield energy at a sample's location after a task transition.
- Energy-guided top-K replay — selecting the highest-energy Task-1 samples — is theoretically and empirically shown to best mitigate forgetting.
- Results validated on Stable Diffusion v1.5 and a pixel-space DDPM fine-tuned on split CIFAR-10, plus domain-incremental rotated MNIST.
- The Hopfield–diffusion connection may extend the framework to transformers and LLMs undergoing continual fine-tuning.
Every time you fine-tune a diffusion model on a new dataset, it quietly starts forgetting the old one. This is catastrophic forgetting , a well-known problem in supervised learning , but for generative models like Stable Diffusion, the mechanism has remained poorly understood. A new paper from researchers at Waseda University and AIST changes that, offering the first principled theoretical explanation of what gets forgotten and why, using a surprising bridge: the energy landscape of modern Hopfield networks.
Why generative forgetting is harder to study
Generative models, including diffusion models, are increasingly used as foundation models and adapted through sequential fine-tuning, making continual learning an essential problem setting. However, continual learning in such generative models remains poorly understood: after a task change, what aspects of the learned distribution are most easily lost, and what replay samples should be prioritized? In supervised learning, forgetting is easy to measure , just check accuracy on old tasks. But diffusion models are unsupervised; there are no class labels to fall back on. You need an internal, model-native metric.
This is the catastrophic forgetting problem , and it has moved from an academic curiosity to a production engineering challenge as models graduate from single-use tools to persistent, multi-task systems. The standard toolkit , regularization, LoRA adapters, replay buffers , is largely empirical. Nobody had a theory for which samples are most at risk.
The Hopfield–diffusion connection
The key insight the paper exploits is a mathematical equivalence that was established in prior work: generative diffusion models can be interpreted as energy-based models, and when trained on discrete patterns, their energy function is asymptotically identical to that of modern Hopfield networks. Modern Hopfield networks (MHNs) , the same family of models behind the attention mechanism in transformers , store memories as attractors in an energy landscape. The energy function is explicitly computable, making it a tractable proxy for the implicit energy inside a diffusion model.
Ambrogioni further made the connection to MHNs explicit by showing that, for finite normalized patterns, the low-noise diffusion energy is asymptotically identical to the MHN energy. The new paper uses this established bridge in a novel setting: continual learning.