University of Tehran's MDP-GRPO Fixes GRPO's Hidden Collapse Under Multi-Constraint Training
MDP-GRPO fixes three fundamental failure modes in GRPO training for instruction-following, boosting strict constraint satisfaction by up to 5% on Llama-3.2-3B
- Problem: Standard GRPO training becomes unstable for multi-constraint instruction following due to discrete, low-variance rewards.
- Three failure modes identified: Low-variance amplification, mean-centering blindness, and zero-variance collapse in z-score group normalization.
- Solution: MDP-GRPO combines multi-temperature sampling, dual-anchor advantages, prospect-theoretic reward shaping, and asymmetric KL regularization.
- Results: Up to 5.0% improvement in strict constraint satisfaction on Llama-3.2-3B across IFEval and FollowBench benchmarks.
- Efficiency: Enables stable training with group size G=4 (vs. standard G=8), reducing compute cost without sacrificing performance.
- General capabilities preserved: MMLU and ARC scores are maintained, confirming no regression on broad knowledge tasks.
Getting a language model to reliably follow multiple constraints at once -- write exactly 200 words, use bullet points, avoid the word 'however', start with a question -- is harder than it looks. Reinforcement learning with verifiable rewards is the natural fit for this problem: each constraint can be checked with a simple rule, giving a clean training signal. But there is a catch. The go-to RL optimizer for LLMs, Group Relative Policy Optimization (GRPO), quietly breaks down in exactly this setting. A new paper from researchers at the University of Tehran introduces MDP-GRPO, a stabilized variant that patches the root causes rather than papering over the symptoms.
Why GRPO struggles with instruction-following
GRPO is an efficient alternative to PPO that sidesteps the need for a separate critic network. Instead of training a value function, GRPO estimates the advantage by generating multiple response candidates for a given input and computing relative rewards by comparing and normalizing each response against others within the same group. The normalization is a simple z-score: subtract the group mean, divide by the group standard deviation.
This works well for math reasoning, where responses vary widely in correctness. But multi-constraint instruction following produces a very different reward landscape. Relying on rule-based checkers provides a highly reliable learning signal and eliminates bias from learned reward models, but the resulting rewards are discrete and frequently low-variance early in training, making stable policy-gradient learning notoriously challenging.
The paper formalizes this into three distinct failure modes of z-score normalization:
- Low-variance amplification: When all responses in a group are nearly identical in quality, tiny random reward differences get magnified by the small standard deviation in the denominator. The model receives huge gradient updates from essentially meaningless signal noise.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.