League of LLMs Makes AI Models Grade Each Other for $1.87
A new evaluation paradigm skips fixed benchmarks entirely, letting eight top models write questions for each other and grade the answers.
Takeaways
- League of LLMs (LOL) is a benchmark-free paradigm where models generate, answer, and mutually score questions across rounds. Paper
- Tested on 8 top models across math and programming, achieving 70.7% Top-k ranking consistency across independent runs.
- Rankings correlate strongly with established benchmarks: ρ=0.93 with LiveCodeBench, ρ=0.74 with MATH500.
- OpenAI models scored their own family 9 points higher than outsiders (p<0.05); DeepSeek showed no such bias.
- Revealed "memorization-based answering" in 7.9% of responses, where models apply templates instead of reasoning.
- Framework is open source; full runs cost ~$2-3 and can be reduced 70% via sparse sampling. Code
Every serious LLM leaderboard has the same dirty secret: the models being tested have probably seen the test. Static benchmarks like MATH500 and HumanEval leak into training corpora, human preference platforms reward emoji-filled fluff, and closed evaluations can't be independently verified. Researchers from the National University of Defense Technology and Chinese Academy of Sciences propose skipping the benchmark entirely and letting the models grade each other.
Their paper, League of LLMs (LOL), organizes multiple models into a self-governed league that competes across multiple rounds. In each round, models take turns writing questions, answering them independently, and mutually scoring each other's work. Final rankings come from aggregating scores across the league.
Why the current evaluation stack is broken
The authors group existing evaluation into three flawed paradigms. White-box evaluations rely on fixed public datasets, so models may memorize test instances through prior training exposure, undermining credibility. Black-box evaluations hide the data but become impossible to verify or reproduce. Human crowdsourced platforms like Chatbot Arena reduce contamination but lean heavily on subjective preferences from users who often lack expert-level domain knowledge.
LOL is designed around four criteria the authors argue any scientific eval should satisfy: dynamic (fresh questions), transparent (fully traceable), objective (no single judge), and professional (expert-level content).