RUMS Beats Models 400x Larger at Picking What AI Should Remember
A new information-theoretic method picks user memory items by how much they sharpen the model's next response, not how similar they look to the query.
Takeaways
- RUMS selects user memory for LLM prompts by measuring mutual information between memory subsets and model outputs.
- Replaces semantic-similarity retrieval, which biases toward redundant, surface-aligned items and misses non-obvious but critical facts.
- Grounded in Bayesian Optimal Experimental Design; entropy reduction proven to correspond to expected utility maximization.
- Matches human selection better than baselines and LLM selectors 400 times larger, with up to 95% cost reduction.
- RUMS-Binary beats ReContriever win rate by roughly 12% on LLaMA 70B; RUMS-Multi beats GPT-4 few-shot by around 13%.
- By design, RUMS can auto-skip personalization entirely when no memory subset meaningfully sharpens the response distribution.
Giving a language model long-term memory of your preferences sounds simple, but the hard part is deciding which remembered facts to actually paste into the prompt. Most systems today grab whatever items look textually similar to the current question. A new paper from Jillian Fisher, Jennifer Neville, and Chan Young Park argues that similarity is the wrong signal, and proposes a method called RUMS that measures how much each memory item changes the model's answer distribution.
Why similarity search misses the point
The standard personalization loop stores user facts (dietary restrictions, writing style, past purchases) in a vector store, retrieves the top-k that match the query, and stuffs them into the prompt. Selection is usually done with BM25, surface-level semantic similarity, or by prompting an LLM to pick.
The authors argue this is fundamentally miscalibrated. Ranking by semantic similarity biases selection toward redundant, surface-aligned items and away from non-obvious but important ones, causing both over-selection and under-selection relative to true utility. The consequences include noisier prompts, degraded responses, higher latency, higher cost, and more privacy exposure. A memory like "I'm allergic to shellfish" may not be textually close to "suggest a restaurant near me," but it should absolutely make it into the prompt.