Skip to main content
Back

Tokyo Researchers' OrderGrad Beats GRPO by Ditching Average Reward Entirely

OrderGrad lets you train models on what you actually care about -- best-of-K, tail risk, or robustness -- not just average reward, with a simple reward reweighting trick.

  • OrderGrad introduces unbiased gradient estimators for L-statistics (weighted sorted rewards), enabling RL beyond mean optimization. [Paper]
  • A single rank-weight vector swap recovers CVaR, VaR, Top-M@K, trimmed mean, median, or best-of-K objectives -- no architecture changes needed.
  • Implemented as a reward transformation compatible with REINFORCE, GRPO, and PPO pipelines out of the box.
  • On LLM math post-training, Top-2@4 outperforms GRPO and MaxPO on pass@k across Qwen2.5-Math-7B and Qwen3-4B-Base benchmarks.
  • Two estimator routes: Likelihood-Ratio (LR) for black-box rewards like LLMs, Reparameterization (RP) for differentiable rewards with lower variance.
  • Open-source MIT library available at github.com/paavo5/ordergrad, requiring only NumPy core with optional PyTorch/JAX backends.

Every policy gradient method in use today -- REINFORCE, GRPO, PPO -- optimizes the same thing: the expected reward. Average performance across all rollouts, averaged across all prompts. That is a reasonable default, but it is often the wrong objective. If you are deploying an LLM that generates K candidate answers and picks the best one, you do not care about the average answer quality. You care about whether at least one answer is great. If you are training a robot for surgery, you do not care about average performance -- you care about the worst-case outcome. Standard policy gradient gives you none of that.

A new paper from researchers at the University of Tokyo introduces OrderGrad, a framework that replaces the standard mean-reward objective with any weighted combination of sorted rewards -- a class of statistics called L-statistics. The result is a plug-and-play reward transformation that slots into existing REINFORCE, GRPO, or PPO pipelines and lets you directly optimize for the distributional property you actually care about.

The gap between training and deployment

The mismatch between training objective and deployment objective is more common than it might seem. Consider a few scenarios:

  • Best-of-K inference: You generate K responses and return the best one. Standard RL trains the model to be good on average, but you want it to reliably produce at least one excellent answer.
  • Risk-sensitive applications: In robotics, finance, or safety-critical systems, a single catastrophic failure can be unacceptable. You want to minimize the expected value of the worst outcomes -- not the mean.
  • Outlier robustness: Noisy reward signals can corrupt training if a few extreme values dominate the gradient. A trimmed mean objective ignores the top and bottom percentiles entirely.

Standard RLVR improves pass@1 performance, but the base model often achieves higher pass@k scores for larger k, and RLVR can narrow the boundary of reasoning capabilities. This is the core tension: optimizing the mean can actively hurt diversity and tail performance.

What are L-statistics, and why do they matter?

An L-statistic (short for linear combination of order statistics) is simply a weighted average of sorted values. Sort your N rewards from lowest to highest, assign a weight to each rank, and take the weighted sum. That is it. The power comes from what different weight vectors recover: