LM Arena's AutoEval Ranks AI Models in Hours, Beating Frontier LLM Judges

The Arena leaderboard has long been the closest thing AI has to an impartial referee. Its scores are grounded in millions of real user preferences, rather than curated test sets that labs can overfit to. The catch: collecting enough human votes to confidently rank a brand-new model takes days. Today, Arena is launching AutoEval, a system that produces a preliminary leaderboard score within hours of a model's release, using a reward model trained on Arena's own human preference data.

The cold-start problem in AI evaluation

Every time a new model drops, there's a frustrating gap between launch and when the Arena leaderboard can say anything meaningful about it. Human votes take time to accumulate, and until enough arrive, confidence intervals are too wide to rank the model reliably. This matters because labs track their position closely: a ranking jump signals competitiveness, and top placements appear in launch blogs, investor materials, and social posts about who is winning frontier AI. AutoEval closes that gap.

How it works

AutoEval is built on a pointwise reward model (RM), a neural network that takes a single prompt-response pair and outputs a scalar score representing how much a human would prefer that response. Unlike a pairwise judge that compares two responses directly, this RM is trained on Arena's large-scale dataset of pairwise human votes, so the scalar scores it produces are calibrated to actual user preferences rather than synthetic labels.

The pipeline runs in three steps:

1. Sample prompts from live Arena evaluations (real user queries, not curated benchmarks)
2. Generate model responses and score each one with the reward model, converting score differences into soft votes
3. Feed those proxy votes through the same Elo-style ranking pipeline used for live human evaluations

The result is an Arena Score that appears directly on the leaderboard, tagged with a lightning bolt icon to mark it as preliminary. Once enough human votes arrive to validate it, the AutoEval score is replaced. Until then, the model gets positioned by its estimated score but receives no official rank.

The numbers behind the claim

Arena's internal validation is compelling. AutoEval's early scores achieved greater than 0.98 ranking correlation with the live scores that followed. Pairwise ordering accuracy scales predictably with how far apart two models actually are:

• When two models are separated by at least 10 Arena points, AutoEval picks the right one in more than 90% of cases
• For gaps above 15 points, it correctly ordered every tested pair
• The text reward model predicts human preferences 8 to 10% more accurately than frontier LLM judges, including Gemini-3-flash/pro and GPT-5

That last point is worth dwelling on. Using a large frontier model as a judge is the dominant approach in automated evaluation today, yet it appears to be a worse approximation of human preference than a purpose-built reward model. Many Arena prompts are hard and contested, with competing models producing responses that each win on different dimensions. In those cases, LLM judges struggle to make the final call, while the reward model still captures what humans actually prefer.

Training a reward model that doesn't go stale

Human preferences shift as models improve and use cases evolve, which makes keeping the RM current a genuine engineering challenge. Arena identifies four specific problems they had to solve:

• Preference drift: Historical data provides breadth and stability, but recent votes reflect current expectations. The RM must balance both without overfitting to either.
• Increasingly subtle differences: As frontier models converge in quality, the gaps the RM needs to detect get smaller and harder to distinguish.
• Ambiguous tie votes: Real users often can't decide between two strong responses. The model needs to handle soft preferences, not just clear wins.
• Temporal holdout validation: To evaluate the RM fairly, Arena trains on historical votes and tests on models released later, a stricter test than random train/test splits.

Beyond text: one method, many modalities

AutoEval covers more than text. Arena trains modality-specific reward models for vision, image generation, and code using the same approach. The image generation RM is particularly notable: trained on more than 3 million preference pairs, it achieves state-of-the-art performance among pointwise reward models on the MMRB2 benchmark, the standard benchmark for evaluating multimodal reward models across tasks like text-to-image, image editing, and visual reasoning.

On MMRB2, Arena's image generation reward model scores 69.7% accuracy, more than 9 points ahead of the next-best baseline (HPSv3 at 60.2%). AutoEval preliminary scores will appear across all major leaderboard categories: Text, Vision, Image Generation, and Code Arena.

What this changes for the field

The conventional wisdom in AI evaluation has been that human preference data is the gold standard but doesn't scale, while automated judges scale but introduce their own biases. AutoEval suggests a third path: a reward model trained specifically on human preference data from a live service can outperform both approaches on the metrics that matter most for leaderboard ranking.

For labs, this means a credible, public signal on a new model within hours of release rather than days. For the broader community, it means the leaderboard stays informative during the frantic first days after a major model launch, exactly when everyone wants to know where it stands. AutoEval scores are live now on the Arena leaderboard and free to view.