Proximal KL Beats GRPO by 2.58% When AI Training Data Goes Stale
A new RLVR training method called Proximal KL swaps PPO-style clipping for bounded future-KL credit, beating GRPO by up to 2.58% on math benchmarks.
- New method Proximal KL replaces PPO clipping with bounded future-KL credit for off-policy RLVR training.
- Beats GRPO by up to 2.58% micro-averaged across eight math benchmarks on Qwen3-1.7B and Qwen3-4B.
- Proves token-wise update shares the same unique stationary policy as exact proximal optimizer under coverage conditions.
- Avoids training collapses seen in plain policy gradient and CISPO during periodic rollout-refresh experiments.
- Separates proximal reference from sampling policy and requires no fitted critic or full-response importance weights.
- Finite future horizon reduces correction variance and improves AIME24 mean@16 and pass@16 scores.
Proximal KL tackles off-policy drift in RLVR
Reinforcement learning with verifiable rewards, or RLVR, trains models against automatically checked outcomes such as correct math answers or passing code. Common training pipelines generate responses asynchronously, reuse batches across optimizer steps, and run inference at lower precision than training. Those choices make the rollouts off-policy because the policy that produced them differs from the policy receiving the update.
That mismatch can bias methods including GRPO, DAPO, and CISPO as rollouts grow stale. A preprint titled Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization proposes a bounded future-KL correction that preserves inexpensive token-wise importance sampling while targeting the stationary policy of an exact proximal update. The paper evaluates the method on Qwen3 models and eight math benchmarks.
Token ratios lose the trail
Off-policy policy gradients typically correct for distribution mismatch with a likelihood ratio between the current policy and the behavior policy that generated the data. Multiplying those ratios across a long response can produce extreme variance because every token contributes another factor. RLVR implementations therefore often apply a separate ratio at each token.
GRPO estimates advantages by comparing responses within a sampled group and applies PPO-style clipping to each token ratio. DAPO uses a related token-wise correction, while CISPO clips the importance weight directly. These bounds stabilize optimization, but they truncate parts of the gradient when ratios cross their thresholds, changing the expected update under stale data.
| Method | Off-policy correction | Behavior under stale rollouts |
|---|---|---|
| GRPO and DAPO | Token-wise ratios with PPO-style clipping | Clipped branches can lose gradient contribution |
| CISPO | Direct clipping of importance weights | Large corrections remain bounded but truncated |
| Proximal KL | Token-wise ratios with bounded future-KL feedback | A finite horizon controls variance without PPO’s hard gate |
A bounded look ahead
Proximal KL assigns each sampled token a correction based on the divergence between the current policy and a proximal reference. The correction includes the current position and as many as
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
