Fudan's Nemori Beats Full Context Memory While Using 88% Fewer Tokens
A new framework decides what LLM agents should remember based on prediction error rather than hand-crafted heuristics, cutting tokens 88% while beating full-context accuracy.
PRO- Nemori uses prediction error to decide what LLM agents should remember, replacing designer heuristics like importance scores and emotional tags.
- Two cascading modules: episodic integration turns raw messages into narratives, then semantic distillation extracts only what existing knowledge could not predict.
- On LoCoMo, beats full-context prompting (80.8 vs 80.6) while using 88% fewer tokens and cutting latency by 47%.
- On LongMemEval's 105K-token conversations, outperforms full context by 16.7% on gpt-4o-mini using 95 to 96% less context.
- As a front-end for A-MEM and MemoryOS, cuts storage 45 to 64% while slightly improving core accuracy.
- Production implementation open-sourced at github.com/nemori-ai/nemori with async Python, PostgreSQL, and Qdrant backends.
Most memory systems for LLM agents decide what to store using rules the designer wrote by hand: importance scores, emotional tags, or factual templates. A new framework called Nemori throws that playbook out and lets the data itself decide, borrowing an idea from neuroscience: if the agent could already predict what happened, it is not worth remembering.
The approach comes from researchers at Fudan, Shanda Group, Beihang, and Shanghai University of Finance and Economics. On the LoCoMo long-conversation benchmark, Nemori edges out feeding the entire dialogue history into the context window while using roughly one-eighth the tokens. On the much longer LongMemEval benchmark, the gap widens dramatically.
The heuristic trap
Memory systems for LLM agents identify useful experiences through two stages: distillation, which determines the entry form of experiences, and management, which handles ongoing maintenance. Existing systems mostly intervene at the management stage, tracking access frequency, temporal decay, or graph links, or bake designer intuition into distillation.
Heuristics risk introducing subjective bias, which is fatal during distillation because it causes irreversible information distortion. To avoid that distortion, systems tend to over-store, which then amplifies retrieval noise. Once you decide to keep everything just in case, retrieval quality collapses.
Prediction error as a signal
Nemori draws on Predictive Coding Theory from visual neuroscience, which frames the brain as a prediction machine that only propagates surprises upward. Applied to agent memory, prediction error signals information worth retaining; whatever is predictable is redundant.
For each new chunk of conversation, Nemori asks its existing knowledge base to guess what happened. The gap between the guess and what actually happened becomes the semantic memory. If the model could already predict it, there is no reason to write it down.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
