CUHK's EEPO Tricks AI Into Forgetting Itself to Boost Math Reasoning by 33%
A new RLVR method breaks entropy collapse by making models temporarily forget what they just sampled, boosting math reasoning up to 33%.
ProTakeaways
- EEPO adds a sample-then-forget step to GRPO rollouts to prevent entropy collapse in RLVR training. Paper
- Average relative gains over GRPO: 24.3% (Qwen2.5-3B), 33.0% (Llama3.2-3B-Instruct), 10.4% (Qwen3-8B-Base).
- Uses a complementary loss
log(1-p)that penalizes high-probability tokens more than low-probability ones. - Unlearning is entropy-gated and applied as a single momentum-free step to the rollout model only.
- Training time is comparable to vanilla GRPO; higher-temperature baselines were roughly 30% slower.
- Code will be available at github.com/ChanLiang/EEPO.
Reinforcement learning with verifiable rewards has become the default recipe for teaching language models to reason, and it has a well-known failure mode: the policy quickly convinces itself that one solution style is best, stops exploring alternatives, and plateaus. A team from CUHK, Microsoft Research Asia, HKBU, and LMU Munich has a strikingly simple fix. Their method, called EEPO (Exploration-Enhanced Policy Optimization), forces the model to briefly forget its own recent answers mid-rollout so it is pushed to try something different.
The exploitation trap in RLVR
Reinforcement learning with verifiable rewards (RLVR) replaces subjective preference feedback with automatically verifiable, rule-based rewards, enabling scalable optimization in domains with reliable checkers such as mathematics and programming. The dominant algorithm powering this wave, GRPO, is efficient but exploitative by nature. Policies over-emphasize exploitation of high-reward trajectories, leading to entropy collapse and reduced final performance.
Training a Qwen2.5-3B with GRPO on MATH, the authors observed a telling divergence: as entropy declines sharply, in-distribution test accuracy continues to rise, whereas performance on out-of-distribution benchmarks such as AMC 2023 deteriorates. The model is memorizing one way of solving problems and losing the ability to generalize.
The authors trace this to a self-reinforcing loop. Several plausible reasoning behaviors exist for a given question, but one mode receives more probability mass. If rollouts predominantly sample this dominant mode and receive positive feedback, the policy further amplifies it while suppressing alternatives. Standard fixes like raising temperature or adding entropy regularization only flatten the distribution uniformly, so the dominant mode is still the most likely to be sampled.