ProactAgent Teaches AI to Decide When Memory Helps, Cutting Token Use in Half
ProactAgent teaches LLM agents to decide when to consult memory using reinforcement learning, cutting interaction steps by 33% and beating all baselines on three benchmarks
Takeaways
- ProactAgent teaches LLM agents to decide when to query memory using RL, rather than retrieving on a fixed schedule.
- The paired-branch trick forks trajectories at retrieval steps to generate counterfactual continuations, giving step-level reward signals for each retrieval decision.
- A structured Experience Base stores five typed entries (factual, episodic, success skills, failure skills, comparative skills) for balanced retrieval.
- ProactAgent hits 73.50% on SciWorld and 71.28% on AlfWorld, with 33% fewer interaction rounds than the best baseline.
- A 3B model with ProactAgent nearly matches a 7B passive-retrieval baseline while using 48% fewer steps -- proactive retrieval partially offsets model size.
- Code is forthcoming on GitHub; paper: arxiv.org/abs/2604.20572
Every time an AI agent gets stuck on a task, it faces a quiet dilemma: should it dig through its memory for a past solution, or just keep trying? Most systems today answer that question with a hard-coded rule -- retrieve at the start of a task, or retrieve after every single step. A new paper from East China Normal University and Shanghai AI Laboratory argues that both approaches are wrong, and proposes a framework called ProactAgent that teaches agents to make that call themselves, on the fly, using reinforcement learning.
The retrieval timing problem
The broader goal here is online lifelong learning -- the idea that an agent should keep getting better as it completes more tasks, without being retrained from scratch. The standard tool for this is an external memory store: log past interactions, retrieve relevant ones at decision time, and let the agent benefit from its own history.
The catch is when to retrieve. The field has converged on three approaches, all of which have serious drawbacks:
- Static initialization: inject memory once at the start of a task. Works fine until conditions shift mid-episode and the agent can't ask for new information.
- Continuous retrieval: query the memory store at every single step. Guarantees access but floods the context window with redundant results and burns tokens.
- LLM-gated retrieval: use a separate model to decide whether to retrieve. More flexible, but adds latency and inference cost on every step.
Existing methods treat retrieval as a passive operation, triggering it only at task initialization or after completing a step. Consequently, agents often fail to identify knowledge gaps during interaction and proactively retrieve the most useful experience for the current decision. ProactAgent's answer is to make retrieval a first-class action that the agent's own policy learns to invoke -- and learns to skip.

Two interlocking engines
ProactAgent is built from two components that reinforce each other. The first is ExpOnEvo (Experience-Enhanced Online Evolution), which manages a structured memory store called the Experience Base. Rather than dumping everything into one flat pool, the experience base organizes historical interactions into typed repositories, including factual memory, episodic memory, and behavioral skills, so that retrieval can provide both relevant evidence and actionable guidance.