Skip to main content
Back

HKUST's GTR Fixes PPO's Blind Spot, Beating It by 25% on Evolving Tasks

GTR replaces PPO's rigid trust region with a Gaussian-shaped constraint that lets agents break out of stale behaviors in continual and non-stationary RL settings

Paper, Full Access
  • PPO's core flaw in continual RL: Standard PPO makes persistent but directionless local updates, causing policy collapse when tasks change -- not due to clipping or model capacity.
  • GTR fix: Replaces the monotonic KL divergence penalty with a Gaussian-shaped trust region that is stiff near the reference policy but releases under sustained high-advantage updates.
  • Mixture Gaussian Anchor: Averages the trust-region constraint across multiple recent policy snapshots to prevent reference staleness during multi-epoch PPO updates.
  • Results: Up to 25% better peak performance on robotic control curricula; enables behavioral mode transitions in 1B-step Craftax training; beats KL-based GRPO on all four math reasoning benchmarks with Qwen3-1.7B.
  • Architecture-agnostic: Works as a drop-in loss term with MLPs, RNNs, SimBa, and Transformers -- no architectural changes required.
  • Limitations: Not yet validated at large LLM scale (>1.7B); lacks a complete theoretical convergence proof for non-monotonic trust regions in non-stationary MDPs.

Proximal Policy Optimization has been the workhorse of modern reinforcement learning, powering everything from robotic locomotion to LLM post-training via RLHF and GRPO. But a new paper from researchers at HKUST, Mila, and Université de Montréal argues that PPO carries a structural flaw that only reveals itself in the wild: when the task changes, PPO gets stuck. Not because it lacks capacity, and not because its clipping range is too tight -- but because it has no sense of direction.

The trap PPO falls into

In standard RL, the environment is assumed to be stationary -- the rules don't change mid-game. PPO's success largely stems from constraining policy updates to remain close to a reference policy via a clipping mechanism, ensuring stable improvement under advantage-driven optimization. That works beautifully when you just need to incrementally refine a behavior. But in continual and non-stationary settings -- where tasks evolve, skills must be unlearned, and new behavioral modes must be adopted -- this same property becomes a liability.

When tasks or required skills evolve over time, effective learning requires not only stable local improvement, but also the ability to leave outdated behavioral modes and transition toward newly advantageous ones. The paper shows this failure empirically: a policy trained sequentially on progressively harder levels of the Procgen Starpilot game collapses after the first task switch, even though PPO can solve each level independently from scratch.

PPO performance collapse during sequential training across difficulty levels

The natural first hypothesis is that the clipping range is too small. The researchers tested this directly. Relaxing the clipping bound and considering schedules where the clipping range grows over time neither modification recovers continual learning ability, indicating that clipping is not the root cause. They also tried refreshing network capacity via stochastic perturbations -- same result. The model collapses regardless.

The real culprit is subtler. PPO performs persistent, directionally inefficient local updates -- the policy is actively updating and exploring in parameter space, but these updates fail to produce consistent improvement, indicating that exploration is directionally inefficient despite being persistent. In other words, the agent is spinning its wheels. It's making updates, but those updates have no geometric sense of which direction leads to a better behavioral mode.

Why geometry matters in policy optimization