Skip to main content
Back

Google and Cornell's Sleep Paradigm Lets LLMs Learn Without Forgetting

Researchers from Google Research and Cornell University propose a 'Sleep' paradigm that lets LLMs consolidate short-term context into long-term parameters — without catastrophic forgetting

Paper, Full Access
  • Researchers from Google Research and Cornell University propose a "Sleep" paradigm for LLMs that consolidates short-term in-context knowledge into stable long-term weights, inspired by human memory consolidation during sleep.
  • Sleep has two stages: Memory Consolidation (distilling high-frequency layers into newly expanded low-frequency parameters) and Dreaming (RL-driven self-generated synthetic training data).
  • On mathematical reasoning benchmarks (AIME-24/25, HMMT-25), Sleep outperforms GRPO and SFT on both Qwen3-1.7B and Qwen3-8B, and is up to 4.8x more compute-efficient than SFT at matched performance.
  • On BABILong, the Sleep-augmented Hope model achieves near-perfect scores at 10M token contexts, where GPT-4 fails beyond 256K tokens.
  • On few-shot ARC abstract reasoning, Sleep achieves 80% success vs. 72.5% for SEAL and 0% for standard ICL.
  • No public code or weights are released; the framework requires the Hope/Nested Learning architecture and is currently a research proof-of-concept. Read the paper.

Every time you fine-tune a language model on new information, you risk erasing what it already knew. This is the core tension in continual learning: how do you teach an AI new tricks without making it forget the old ones? A new paper from researchers at Google Research and Cornell University, titled "Language Models Need Sleep," proposes a biologically inspired answer , give the model a sleep cycle.

The work introduces a framework called the Sleep paradigm, which draws a direct analogy to how the human brain consolidates memories during sleep. The key insight is that current LLMs are stuck in a kind of anterograde amnesia: they can learn from context within a session, but that knowledge evaporates the moment the context window closes. Nothing makes it into the model's weights permanently without expensive retraining.

The Problem With Always Being Awake

Most approaches to keeping LLMs up-to-date fall into two painful camps:

  • Re-pretraining on an expanded dataset , effective but prohibitively expensive for frequent updates.
  • Continual fine-tuning via methods like LoRA , cheap, but prone to catastrophic forgetting (CF), where learning new tasks overwrites the weights responsible for old ones.

Existing models lack the ability to continually learn and effectively transfer their temporal in-context knowledge to their long-term parameters. The researchers frame this as a memory architecture problem: transformer architectures can be interpreted as two-level memory systems, in which the attention module acts as short-term memory (updated every context window) and MLP layers act as long-term memory (frozen after pretraining). There is nothing in between , no mechanism for gradually promoting short-term knowledge into stable long-term storage.

Diagram comparing conventional machine learning vs. continual learning paradigms with active/wake and sleep phases

Two Stages of Sleep

The Sleep paradigm replaces the conventional train/test split with two recurring phases: an active (wake) phase, where the model processes new input normally, and a sleep phase, where it stops receiving external data and focuses entirely on internal consolidation. Sleep itself has two sub-stages, mirroring NREM and REM sleep in humans.