Google DeepMind's Gemma Cuts RLHF Training Data Needs by 10x
Google DeepMind shows an online RLHF algorithm with uncertainty-guided exploration hits offline RLHF's quality using 10x less human feedback, projected to reach 1,000x.
ProTakeaways
- Google DeepMind's Efficient Exploration at Scale reports a 10x data efficiency gain over offline RLHF.
- The algorithm reaches offline RLHF's 200K-label quality using fewer than 20K labels on Gemma 9B.
- Extrapolated curves project a 1,000x efficiency gain at the one-million-label scale.
- Three ingredients: an affirmative nudge that prevents tanking, an epistemic neural network, and information-directed query selection.
- Reward model uncertainty comes from an ensemble of 100 prior and 100 differential MLP heads, adding under 5% parameters.
- No code or weights released; feedback is simulated by a Gemini 1.5 Pro reward model rather than real humans.
A team at Google DeepMind has published results suggesting that most of the human feedback poured into aligning large language models is being wasted. Their new online RLHF algorithm reaches the same win rate as a standard offline pipeline while using roughly a tenth of the preference labels, and the gap widens as data scales up.
The paper, titled Efficient Exploration at Scale, comes out of what the authors call the Efficient Agent Team. With Gemma large language models, their algorithm matches the performance of offline RLHF trained on 200K labels using fewer than 20K labels, a more than 10x gain in data efficiency. Extrapolating the curve further, they project a 1,000x gain at the million-label scale.
Why RLHF has a data problem
Reinforcement learning from human feedback is the step that turns a raw pretrained model into something that follows instructions and behaves. In the standard offline recipe, you collect a big pile of human preference comparisons up front, fit a reward model, then optimize the policy against it. The obvious weakness: that pile is fixed, generated by a policy that will soon be obsolete, and nothing about it is chosen to be informative.
Recent work suggests current RLHF techniques scale poorly, yielding only marginal performance gains even when preference data is substantially increased. That is the backdrop for this paper. Throwing more labels at offline RLHF barely moves the needle, so the question becomes whether smarter data collection can.
Earlier online and active-exploration methods have reported gains, but modest ones. The prior literature reports improvements of 2x to 5x and works with a much more limited range of prompts than the diverse set used here. A 10x-to-1,000x jump would be a different animal.
Three ingredients that make it work
The core idea is to update the reward model and language model incrementally as choices arrive, and to pick the response pairs the human sees so they resolve the most uncertainty. Three innovations enable the gains: a small affirmative nudge added to each reinforcement signal, an epistemic neural network that models reward uncertainty, and information-directed exploration.