How much does a language model actually remember? It sounds like a simple question, but it has resisted clean answers for years. Extraction attacks, membership inference, differential privacy bounds , all useful tools, but none of them directly answer the core question: how many bits of specific training data are locked inside a model's weights? A new ICML paper from researchers at Meta, Google DeepMind, Cornell, and NVIDIA finally puts a number on it.
The measurement problem nobody solved
The field has long struggled with a fundamental confusion: when a model outputs something that was in its training set, is that memorization or generalization? Language models can be coerced to output almost any string, so the fact that a model outputs something is not necessarily a sign of memorization. A model that correctly answers a math problem it was trained on might simply be doing math , not recalling a specific example.
Prior studies of language model memorization struggled to disentangle memorization from generalization. The paper formally separates memorization into two components: unintended memorization (the information a model contains about a specific dataset) and generalization (the information a model contains about the true data-generation process). This distinction is what makes the measurement possible.
Compression as the measuring stick
The key insight is borrowed from information theory: if a model helps you compress a dataset, it must contain information about that dataset. The authors use Kolmogorov complexity , the theoretical minimum description length of a piece of data , as the formal backbone, then approximate it in practice using the model's own log-likelihoods. Concretely, unintended memorization of a sample is measured as how many fewer bits you need to encode it when the trained model is available as a reference, compared to using only a baseline model trained on the full distribution.
To isolate pure memorization capacity, the team first eliminated generalization entirely by training on datasets of
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves