EasyPPO traces PPO collapses to the critic

Researchers at UC Berkeley and collaborating institutions report in the EasyPPO paper that unstable value learning drives PPO failures across three large language model reinforcement learning settings. Their method stabilizes the critic with three targeted changes while preserving PPO’s policy objective and actor optimizer.

PPO remains common in LLM post-training because it can learn from continuous, binary, and multi-step rewards. Its reliability depends on a critic whose predictions shape every policy update, so critic errors can stall training or erase gains from an otherwise healthy run. EasyPPO addresses that failure point without introducing a new reinforcement learning algorithm.

How critic errors reach the policy

PPO trains an actor alongside a critic. The actor generates responses, while the critic estimates the expected future return, or cumulative reward, for each prompt and partial response. PPO combines observed returns with those estimates to calculate advantages, which determine how strongly each generated token should be reinforced.

Inaccurate value estimates produce noisy advantages. The actor then strengthens or weakens tokens using a distorted learning signal, allowing critic instability to spread into the policy even when the policy loss and optimizer behave as designed.

Two failure modes feed collapse

Truncation disappears from training

A filtering practice common in GRPO implementations drops rollouts that reach the token limit before completion. Applying that filter to both the actor and critic conditions training on completed responses: the critic never learns that exhausting the context budget is undesirable, and the reported reward excludes the failed, truncated trajectories.