HKUST's CHARM Catches Reward Models Quietly Favoring Certain AI Outputs
New research uncovers that reward models unfairly favor certain policy models, then uses Chatbot Arena Elo ratings to fix the bias.
ProTakeaways
- CHARM identifies model preference bias where reward models over-score certain policy models regardless of actual quality.
- The method uses Chatbot Arena Elo ratings to derive expected win rates and calibrate reward model scores.
- Skywork-RM gained 5.2 points on RM-Bench Chat and 1.6 average after CHARM calibration.
- DPO with CHARM-calibrated rewards improved Qwen2.5-7B on IFEval by up to 2.77 points.
- Calibration flipped reward model preference for emojis from 2.38x to 0.61x, exposing hidden style bias.
- Code released at github.com/HexagonStar/CHARM and works with any scalar reward model.
Reward models are the invisible judges of modern language model training. They score outputs, decide which responses win, and shape what a policy model learns during reinforcement learning from human feedback. New research from a team spanning HKUST(Guangzhou), Alibaba, Queen's University Belfast, and the University of Zurich shows those judges are quietly playing favorites, and proposes a fix built on Chatbot Arena rankings.
The paper introduces CHARM (Chatbot Arena calibrated Reward Modeling), a calibration method that identifies and corrects a subtle bias the authors call model preference bias. Reward models systematically assign disproportionately high scores to responses from certain policy models, leading to unfair judgments. Some policy models get inflated scores that have nothing to do with whether their answers are actually better.
The judges have favorites
The team started with a reasonable assumption: if a reward model truly reflects human preferences, its scores should correlate with Chatbot Arena Elo ratings, which are computed from millions of blind pairwise votes by real users. They tested five popular reward models (Skywork-Reward-Llama, RM-Mistral-7B, FsfairX-LLaMA3-RM, GRM-llama3-8B-distill, and BTRM-Qwen2-7b) against a diverse set of policy models on AlpacaEval.

The correlation mostly held, with glaring outliers. Some policy models, such as Gemma-2-9b-it-SimPO, receive disproportionately high RM scores, sometimes surpassing significantly stronger models based on Elo rankings. A similar trend appears in the AlpacaEval leaderboard, suggesting the bias may also exist in LLM-as-a-judge systems. The overvalued models had one thing in common: they undergo preference optimization, implying a potential source of systematic bias.