Skip to main content
Back

USC's SHRED Erases What AI Memorized Without Destroying Everything Else

SHRED erases memorized content from LLMs without a retain set, outperforming methods that need curated data to stay coherent

Paper, Full Access
  • SHRED is a new LLM unlearning method that removes memorized content without needing a retain set of curated examples.
  • It identifies which tokens carry memorized knowledge using Shannon surprisal (low-probability = high-information = memorized), then selectively demotes only those token logits.
  • The model uses its own output distribution as a self-distillation teacher, making the process data-efficient and operationally simple.
  • SHRED sets a new Pareto-optimal tradeoff between forget efficacy and model utility across four benchmarks: TOFU, MUSE, RWKU, and Hubble.
  • It is robust against relearning attacks and membership-inference attacks, and stays stable across many sequential unlearning rounds.
  • No code has been released yet; the method is described in full in the paper.

Every time someone invokes the GDPR's "right to be forgotten," or a copyright holder demands their text be scrubbed from a deployed model, an AI team faces the same uncomfortable question: how do you surgically remove what a model has memorized without destroying everything else it knows? SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion), a new method from researchers at USC, proposes a cleaner answer -- and does it without the extra data dependency that has quietly plagued the field.

The retain-set trap

Machine unlearning for LLMs aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining. Most existing methods require a retain set of curated examples to prevent catastrophic degradation of general model utility, creating an extra data dependency that complicates deployment. Think of the retain set as a safety net: while the model is being trained to forget a document, you simultaneously feed it examples of things it should not forget, anchoring its general capabilities.

The problem is that assembling that safety net is expensive and legally fraught. You need a representative sample of benign data, you need to store it, and in regulated environments you may need to justify why you're using it. Machine unlearning for LLMs is a rapidly evolving area targeting the selective removal of specific knowledge from a deployed model without full retraining, with principal motivations including regulatory compliance (GDPR "right to be forgotten"), privacy, copyright enforcement, safety, and bias mitigation. The retain-set requirement adds friction to every one of those use cases.

The key insight: not all tokens are equal

SHRED's core observation is deceptively simple. When a language model has memorized a piece of text, the memorization is not spread evenly across every word. An LLM that memorizes forget-set samples assigns low probability to the high-information tokens -- such as names, relations, and actions -- and high probability to low-information tokens, such as linguistic structure and commonsense knowledge. In other words, the model is surprised by the specific facts (a person's birthday, a book's exact phrasing) but completely unsurprised by the grammar holding them together.