Skip to main content
Back

Waseda Researchers Explain Why Stable Diffusion Forgets Using Hopfield Energy

A new theoretical framework uses Hopfield energy to predict which samples diffusion models forget first — and which to replay to prevent it

Paper, Full Access
  • Researchers prove that high-energy, outlier-like samples in diffusion models are forgotten more than typical cluster samples after task changes.
  • The framework uses modern Hopfield network energy as a tractable, model-native proxy for forgetting in diffusion models.
  • A new concept, "intrinsic forgetting," is defined as the increase in Hopfield energy at a sample's location after a task transition.
  • Energy-guided top-K replay — selecting the highest-energy Task-1 samples — is theoretically and empirically shown to best mitigate forgetting.
  • Results validated on Stable Diffusion v1.5 and a pixel-space DDPM fine-tuned on split CIFAR-10, plus domain-incremental rotated MNIST.
  • The Hopfield–diffusion connection may extend the framework to transformers and LLMs undergoing continual fine-tuning.

Every time you fine-tune a diffusion model on a new dataset, it quietly starts forgetting the old one. This is catastrophic forgetting , a well-known problem in supervised learning , but for generative models like Stable Diffusion, the mechanism has remained poorly understood. A new paper from researchers at Waseda University and AIST changes that, offering the first principled theoretical explanation of what gets forgotten and why, using a surprising bridge: the energy landscape of modern Hopfield networks.

Why generative forgetting is harder to study

Generative models, including diffusion models, are increasingly used as foundation models and adapted through sequential fine-tuning, making continual learning an essential problem setting. However, continual learning in such generative models remains poorly understood: after a task change, what aspects of the learned distribution are most easily lost, and what replay samples should be prioritized? In supervised learning, forgetting is easy to measure , just check accuracy on old tasks. But diffusion models are unsupervised; there are no class labels to fall back on. You need an internal, model-native metric.

This is the catastrophic forgetting problem , and it has moved from an academic curiosity to a production engineering challenge as models graduate from single-use tools to persistent, multi-task systems. The standard toolkit , regularization, LoRA adapters, replay buffers , is largely empirical. Nobody had a theory for which samples are most at risk.

The Hopfield–diffusion connection

The key insight the paper exploits is a mathematical equivalence that was established in prior work: generative diffusion models can be interpreted as energy-based models, and when trained on discrete patterns, their energy function is asymptotically identical to that of modern Hopfield networks. Modern Hopfield networks (MHNs) , the same family of models behind the attention mechanism in transformers , store memories as attractors in an energy landscape. The energy function is explicitly computable, making it a tractable proxy for the implicit energy inside a diffusion model.

Ambrogioni further made the connection to MHNs explicit by showing that, for finite normalized patterns, the low-noise diffusion energy is asymptotically identical to the MHN energy. The new paper uses this established bridge in a novel setting: continual learning.