Northwestern Researchers Expose a Hidden RL Training Flaw Doubling Agent Success Rates
RAGEN-2 exposes a silent RL training failure where agents produce fluent but input-agnostic reasoning, invisible to entropy metrics, and fixes it with a lightweight reward-variance filter.
Takeaways
- RAGEN-2 identifies template collapse: RL-trained agents produce fluent but input-agnostic reasoning even when entropy looks healthy, a failure invisible to all existing metrics.
- The fix is decomposing reasoning quality into two axes: within-input diversity (entropy) and cross-input distinguishability (mutual information), which correlates 2x better with task performance.
- SNR-Aware Filtering skips low-reward-variance prompts each training step, preventing input-agnostic regularization gradients from dominating updates.
- Results span 7 environments across PPO/GRPO/DAPO/Dr.GRPO, Qwen2.5 (0.5B–7B), Llama3.2-3B, and multimodal VL models, with consistent gains.
- The root cause is a gradient SNR imbalance: low reward variance collapses task gradients while KL/entropy regularization remains constant, erasing cross-input differences.
- The paper is accepted as an ICML 2026 Oral; code and project page available at ragen-ai.github.io/v2.
Reinforcement learning for multi-turn LLM agents has a hidden problem that nobody was measuring. RL training of multi-turn LLM agents is inherently unstable, and entropy is widely used to track reasoning stability. But entropy only measures diversity within the same input and cannot tell whether reasoning actually responds to different inputs. A new paper from researchers at Northwestern, Stanford, Microsoft, and several other institutions identifies this blind spot and gives it a name: template collapse.
The ghost in the training loop
Here is the failure mode in plain terms. Imagine an agent trained on a grid puzzle. After many RL updates, it still produces long, varied-looking reasoning chains for each turn. Token-level entropy looks healthy. But if you compare the reasoning across different puzzles, you notice something alarming: the chains are nearly identical. The model has learned a reusable narrative skeleton that satisfies the format and even scores occasional rewards, but has stopped actually reading the input.
When entropy remains high, reasoning can still drift toward fixed templates that appear diverse within any single input but are effectively the same across inputs. This is template collapse, a failure mode invisible to both reward and entropy metrics. This risk is especially acute in multi-turn settings, where sparse rewards cannot distinguish input-driven reasoning from templated reasoning that merely happens to succeed.

Two axes, not one
The core theoretical insight is a decomposition of reasoning diversity into two orthogonal dimensions using Shannon's identity:
H(Z) = I(X;Z) + H(Z|X)
where H(Z|X) is within-input diversity (conditional entropy: how varied are the model's outputs for a single prompt?) and I(X;Z) is cross-input distinguishability (mutual information: how much does knowing the reasoning tell you about which input produced it?). Entropy-based metrics only track the first term. Even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. This is template collapse, a failure mode invisible to entropy and all existing metrics.
The four resulting quadrants are: Diverse Reasoning (high entropy, high MI -- the desired state), Template Collapse (high entropy, low MI -- the dangerous blind spot), Compressed Reasoning