Skip to main content
Back

Tongji University's InfoMem Teaches AI to Remember What Actually Matters

InfoMem introduces an information-theoretic reward that teaches LLMs to build better memory summaries while reading long documents chunk by chunk

  • InfoMem introduces an answer-conditioned information-gain reward for training chunk-wise long-context memory agents, measuring how much a model's final memory increases the log-likelihood of the correct answer.
  • The reward is applied only to successful rollouts, normalized within each group, and conditioned on the ground-truth answer rather than the query, each choice validated by ablation.
  • On four out-of-distribution benchmarks, InfoMem outperforms both outcome-only GRPO and ReMemR1, and is the only method that improves over the base model on all four tasks.
  • A synthetic diagnostic using hallucinated SQuAD contexts shows r_gain achieves MRR 0.977 and SNR 2.96, nearly 5x the signal stability of embedding-based alternatives.
  • Outcome-only GRPO degrades on multi-needle retrieval (MRCR-8needle) from 0.26 to 0.06; InfoMem recovers to 0.28, showing the memory-specific reward prevents retrieval collapse.
  • Code is publicly available; the method is a drop-in reward shaping module for any GRPO-based chunk-wise memory agent.

Getting an LLM to read a 200-document corpus and remember the right things is harder than it sounds. The model has to decide, chunk by chunk, what to keep in a fixed-size memory buffer and what to discard. Train it with only a final-answer reward and it learns to get answers right, but it has no signal for why a particular memory was good or bad. A new paper from Tongji University and Shanghai AI Laboratory introduces InfoMem, a reward mechanism that directly grades the quality of the memory a model builds, not just whether it gets the final answer correct.

The memory problem hiding inside long-context RL

The chunk-wise memory agent paradigm works like this: split a long document into chunks, have the model read each one and update a compact text-based memory, then generate a final answer from that accumulated memory alone. Chunk-wise memory agents address the long-context challenge by sequentially reading document chunks, updating a compact memory, and generating the final answer from the accumulated memory. It is a clean, scalable design that sidesteps the quadratic cost of full-context attention.

The training problem, however, is subtle. Training these agents using reinforcement learning typically relies on a single reward signal, such as the correctness of the final answer. This sparse reward, provided only at the end of the reasoning process, offers limited guidance for the long sequence of intermediate memory updates, leading to inefficient optimization and suboptimal memory management strategies.

The gap is especially sharp when you look at successful rollouts. Multiple attempts may all produce the correct answer, so they all get the same reward of 1. But one memory might contain only the critical supporting sentence, while another is bloated with distractors. The binary outcome reward cannot tell the difference. Existing RL-based chunk-wise agents either rely on sparse final-answer rewards or use lexical intermediate rewards for memory and retrieval actions. These signals supervise task success or local overlap, but do not directly evaluate whether the final memory supports the ground-truth answer.

Measuring memory quality with information gain

InfoMem reframes the question: how much does the final memory actually help the model predict the correct answer? The answer is measured with a simple log-likelihood comparison. The model scores the ground-truth answer twice, once with the final memory in context and once without. The difference in per-token average log-probability is the information-gain reward.