Researchers Rewrite RL's Core Objective to Make AI Diversity Free
A new RL framework replaces scalar rewards with reward distributions, making behavioral diversity a natural byproduct of uncertainty rather than a fragile tuning knob.
Takeaways
- Researchers propose replacing RL's scalar reward with a distribution over reward functions, making behavioral diversity emerge naturally from uncertainty.
- The new objective is a strict generalization of vanilla policy gradient and action-set methods, which fall out as special cases.
- Diversity is controlled by the spread of the reward function distribution -- no separate entropy coefficient or diversity bonus required.
- The framework avoids the classic performance-diversity trade-off: experiments show diverse behavior without sacrificing expected reward.
- Primary use cases include LLM fine-tuning (RLHF), scientific discovery, and any setting where the reward function is ambiguous or imperfect.
- Current work is limited to the contextual bandit setting; no code has been released alongside the paper.
Reinforcement learning has a diversity problem. When you train an agent -- or fine-tune a language model -- with a fixed scalar reward, it converges to a single deterministic policy: the one that scores highest on average. That's fine if the reward function perfectly captures what you want. But in practice, reward functions are imperfect proxies, human preferences are ambiguous, and committing to one behavior can leave a lot of value on the table. A new paper published on arXiv by a team of researchers proposes a fundamental rethink of the RL objective itself to address this.
The problem with patching diversity in
The standard fix for getting diverse behavior out of RL is entropy regularization -- adding a bonus that rewards the policy for being uncertain about its own actions. It works, but it comes with a catch: you're trading off expected reward for stochasticity. Set the entropy coefficient too high and you sacrifice performance; too low and the policy collapses back to a single mode. This is a well-known pain point in LLM fine-tuning, where RLVR still suffers from the exploration-exploitation dilemma, which manifests as a rapid decrease in policy entropy, limiting the performance of LLMs.
Diversity bonuses -- extra reward signals designed to push the agent toward novel states or actions -- have similar issues. They rely on heuristic metrics that can rank policies incorrectly, and they require careful, often brittle hyperparameter tuning. The core argument of this paper is that both approaches are treating a symptom rather than the cause.
Uncertainty as the root cause of diversity
The researchers' central insight is this: