Skip to main content
Back

Nankai's UEC-RL Fixes GRPO Collapse, Boosting Geometry3K Accuracy by 37.9%

A new ACL 2026 method fixes entropy collapse in GRPO by pushing exploration on hard prompts and replaying winning trajectories to stabilize training.

  • UEC-RL is a new RL training method accepted to ACL 2026 Findings, targeting GRPO's entropy collapse.
  • It boosts sampling temperature only on prompts none of the rollouts solved, and replays high-advantage trajectories to consolidate correct reasoning.
  • Achieves 37.9% relative accuracy gain over GRPO on Geometry3K while running at 0.79x per-step cost.
  • Beats GRPO, DAPO, KL-cov, and Entropy-Adv on Qwen2.5-Math-7B, Llama3.1-8B, and Qwen2.5-VL-7B across text and multimodal benchmarks.
  • Improves both Pass@1 and Pass@k, showing preserved policy diversity rather than mode collapse.
  • Open-source Apache 2.0 code available at github.com/597358816/UEC-RL, forked from EasyR1.

Reinforcement learning has become the dominant recipe for teaching large language and vision-language models to reason, but the workhorse algorithm behind DeepSeek-R1 style training, Group Relative Policy Optimization (GRPO), has a known nasty habit: it collapses. Policy entropy drops fast, the model locks in on a narrow set of reasoning patterns, and any deeper trajectory it might have discovered stays permanently out of reach. A new paper from Nankai University, Zhongguancun Academy, and Shanghai Jiao Tong University, accepted to ACL 2026 Findings, proposes a fix called UEC-RL (Unified Entropy Control for RL) that pushes exploration up on hard prompts and pulls it back down once useful behaviors are found.

On the Geometry3K visual math benchmark, UEC-RL delivers a 37.9% relative accuracy improvement over vanilla GRPO while running at roughly 0.79x the per-step cost. The full code is on GitHub under Apache 2.0, forked from the EasyR1 framework.

Why GRPO keeps strangling itself

GRPO drops the critic used in PPO and instead estimates advantages by normalizing rewards within a group of sampled rollouts per prompt. It is efficient, but the authors identify two structural failure modes that show up under identical training settings.

  • Entropy collapse. On text-only reasoning, policy entropy drops rapidly, responses homogenize, and low-probability but informative reasoning chains are never sampled again.
  • Training instability.