LM Arena's AutoEval Ranks AI Models in Hours, Beating Frontier LLM Judges
Arena launches AutoEval, a reward-model-based system that scores new AI models in hours instead of days, with >0.98 correlation to live human votes
- AutoEval launches today: Arena now produces preliminary leaderboard scores for new models in hours, not days, using a reward model trained on millions of real user votes.
- >0.98 ranking correlation: AutoEval scores align extremely closely with final live human vote rankings in temporal holdout tests.
- Beats LLM judges by 8-10%: Arena's text reward model predicts human preferences more accurately than frontier models like GPT-5 and Gemini used as judges.
- Covers all modalities: Separate reward models handle Text, Vision, Image Generation, and Code Arena, each trained on modality-specific preference data.
- SOTA on MMRB2: The image generation reward model scores 69.7% on the MMRB2 benchmark, 9+ points ahead of the next-best baseline.
- Scores are clearly labeled: AutoEval entries appear with a lightning bolt tag and no official rank until live human votes validate them. View them at arena.ai/leaderboard.
The Arena leaderboard has long been the closest thing AI has to an impartial referee. Its scores are grounded in millions of real user preferences, not curated test sets that labs can overfit to. The catch: collecting enough human votes to confidently rank a brand-new model takes days. Today, Arena is launching AutoEval, a system that produces a preliminary leaderboard score within hours of a model's release, using a reward model trained on Arena's own human preference data.
The cold-start problem in AI evaluation
Every time a new model drops, there's a frustrating gap between launch and when the Arena leaderboard can say anything meaningful about it. Human votes take time to accumulate, and until enough arrive, confidence intervals are too wide to rank the model reliably. This matters because labs track their position because a ranking jump signals competitiveness, and top placements appear in launch blogs, investor materials, and social posts about who is winning frontier AI. AutoEval closes that gap.
How it works under the hood
AutoEval is built on a pointwise reward model (RM) -- a neural network that takes a single prompt-response pair and outputs a scalar score representing how much a human would prefer that response. This is different from a pairwise judge that compares two responses directly. Arena trains this RM on its large-scale dataset of pairwise human votes, so the scalar scores it produces are calibrated to actual user preferences, not synthetic labels.