STABLEVAL Fixes the AI Leaderboard Lottery That Flips Rankings
STABLEVAL replaces majority voting with a Bayesian framework that models annotator confusion to produce stable, reproducible AI system rankings
Takeaways
- The problem: Majority vote annotation aggregation produces AI system rankings that flip when annotator subsets change, making evaluations unreproducible.
- The solution: STABLEVAL uses a Bayesian model to infer per-annotator confusion matrices and per-item difficulty, producing calibrated Posterior Expected Credit scores instead of hard labels.
- Key distinction: Unlike Dawid-Skene (which recovers ground-truth labels), STABLEVAL is designed specifically to produce stable AI system rankings under annotator heterogeneity.
- Tested on 4 benchmarks: MT-Bench (LLM preference), ConvAbuse (safety), QAQS (QA quality), and MSLR (medical summarization), consistently outperforming majority vote on ranking stability.
- Formal guarantee: The paper proves majority vote is inherently unstable under disagreement, while STABLEVAL achieves stable rankings in expectation under its generative model.
- No public code yet: The methodology is fully described in the paper but no open-source implementation was released at time of publication.
Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation. If you have ever run a human evaluation study and gotten a different leaderboard ranking the second time around, you have hit this problem firsthand. A new paper introduces STABLEVAL, a framework designed to make those rankings actually mean something.
The leaderboard lottery
AI is currently facing a reproducibility crisis driven by unreliable evaluations and unrepeatable experimental results. While human raters are often used to assess models for utility and safety, they introduce divergent biases and subjective opinions into their annotations. The standard fix is majority vote: collect labels from several annotators, take the most common answer, and move on. Simple, but deeply flawed.
Majority vote discards annotator reliability and item-level ambiguity, often yielding unstable comparisons across annotator subsets. In practice, this means that if you swap out even a few annotators from your pool, the ranking of your AI systems can flip entirely. Standard evaluation practices typically rely on a small number of annotations per item, often 3 to 5, and lack the persistent rater identifiers necessary to model individual variance across items. The result is that a model that "wins" one evaluation round might "lose" the next, not because it got worse, but because the annotator sample changed.
Majority voting and averaging are common approaches used to resolve annotator disagreements and derive single ground truth labels from multiple annotations. However, annotators may systematically disagree with one another, often reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. This is not noise to be averaged away. It is signal.