Epoch Rebuilds Its ECI Leaderboard With Bayesian Stats to Sharpen AI Rankings

Epoch AI has replaced the statistical model behind its ECI leaderboard, which combines dozens of AI benchmarks into one capability score per model. The Epoch Capabilities Index still spans roughly 0 to 170, with Claude 3.5 Sonnet anchored at 130 and GPT-5 at 150. The revised method preserves that scale while producing posterior distributions that support direct probability statements about model differences.

Frontier rankings remain almost unchanged, while uncertainty estimates become narrower and better calibrated. Scores for weaker models fall substantially because Epoch removed a regularization term that had pulled sparse results toward the average.

One score from uneven tests

ECI uses Item Response Theory, described in Epoch’s methodology paper, to combine benchmarks with different difficulty levels and scoring behavior. Each benchmark receives a difficulty and slope, each model receives a latent capability value, and a logistic function converts those parameters into a predicted score between 0% and 100%.

The previous implementation estimated capability, difficulty, and slope by minimizing squared error against observed benchmark scores. A regularization term stabilized models with little data, especially weak models, by pulling their estimates toward the mean.

Epoch estimated uncertainty through bootstrap resampling. Each model’s benchmark results were sampled with replacement, and the full model was refitted for every sample. The median model has ten recorded scores, while some have only four. A bootstrap sample omits about one-third of the original observations, making the resulting spread unstable for sparsely evaluated models. Separate resampling also failed to capture shared uncertainty between models, which limited reliable head-to-head comparisons.

From point estimates to posteriors

The new implementation uses Markov chain Monte Carlo through PyMC’s No-U-Turn Sampler. MCMC generates many plausible parameter sets from the posterior distribution, which represents parameter uncertainty after combining the observed scores with the model’s prior assumptions. Each model’s ECI is therefore represented by a distribution rather than a single fitted value.

• A beta likelihood models bounded scores. The beta distribution assigns probability only between 0 and 1 and allows score variance to shrink near either boundary. A constant-variance normal model can assign probability to impossible values below 0% or above 100%.
• Each benchmark gets its own noise estimate. At a predicted score of 50%, results typically range from 46% to 54% on METR Time Horizons and from 32% to 68% on VPCT. The model learns a separate sigma_b parameter for each benchmark.
• All benchmarks help define the scale. The previous fit fixed WinoGrande’s slope at 1. The Bayesian version normalizes slopes across the benchmark set, reducing dependence on one benchmark’s behavior.
• Priors replace the old regularizer. Capability and difficulty receive flat priors. Benchmark slopes and noise use hierarchical log-normal priors, allowing related parameters to share statistical strength while retaining benchmark-specific values.
• Joint estimation preserves correlations. Every model and benchmark parameter is sampled together, so comparisons retain the uncertainty shared across models.

Frontier order stays put

Across the 176 models with an ECI of at least 125, the old and new estimates have an R² of 0.995 and differ by 1.1 points in root mean squared error. Every revised score falls within its previous 90% confidence interval, while the new 90% intervals are 13% narrower on average.

Estimated progress rates also remain close. The pre-reasoning trend from GPT-4 through GPT-4.5 rises from 6.2 to 8.0 ECI points per year. The reasoning-model trend beginning with o1-mini increases from 15.7 to 16.6 points per year. Much of the change comes from a modest expansion of the scale between the two anchor models.

All 76 models below ECI 125 move downward, by 13.7 points on average. The 21 models previously below 100 decline by an average of 27 points. Removing the old regularization term accounts for most of this shift because weak models with few recorded scores no longer receive the same pull toward the overall mean.

Pairwise odds get sharper

Joint sampling makes uncertainty in model gaps smaller than uncertainty in either standalone score because shared sources of variation cancel during comparison. The 90% credible interval for the difference between GPT-6 Astra and Claude Opus 5.5 runs from minus 0.9 to plus 2.1 ECI points, roughly half the width of either model’s individual interval.

Epoch can calculate the probability that one model exceeds another by counting the share of posterior samples in which its capability value is higher. Under this model, GPT-6 Astra has a 72% probability of exceeding Claude Opus 5.5 and a 99% probability of exceeding Claude Fable 5.1.

These figures are conditional on ECI’s benchmark selection, data, and statistical assumptions. Production choices still require evaluations that reflect the application’s tasks, latency limits, cost constraints, and failure tolerance.

Held-out scores meet the target

Bayesian predictive intervals contain 89.6% of held-out benchmark scores against a 90% target, compared with 84.2% under the frequentist fit. In-sample accuracy is similar across the two methods, while leave-one-out root mean squared error improves slightly under the Bayesian model. Those results indicate better uncertainty calibration and a modest gain when predicting unseen scores.

Epoch also tested alternative likelihoods, including censored normal variants that handle exact zero scores more directly. Those variants produced poorer calibration elsewhere, leading the organization to retain the beta likelihood.

What ECI users should update

• Refresh stored scores before analysis. Methodology changes materially affect models below ECI 125, so values from different versions should carry a methodology date or version.
• Use pairwise probabilities for close comparisons. Standalone intervals omit correlations that the joint posterior captures.
• Preserve task-specific evaluations. ECI measures broad benchmark capability and does not encode workload-specific cost, latency, reliability, or safety requirements.

The revised scores, underlying data, and code are available on the ECI page. Existing frontier analyses should produce similar rankings, while low-end comparisons and uncertainty-sensitive research may change substantially under the new model.