ScoringBench redraws tabular model rankings with probabilistic scores

Tabular foundation models such as TabPFN and TabICL can produce a predictive distribution for each input, assigning probabilities across possible target values. Most regression benchmarks reduce that output to a single estimate and grade it with root-mean-square error or R². The ScoringBench paper evaluates the entire distribution and finds substantial changes in model rankings.

Point metrics leave uncertainty ungraded

Root-mean-square error measures the distance between a point prediction and the observed value, while R² measures how much target variation those predictions explain. Coverage, calibration, interval width, and tail probabilities remain unmeasured. A predictive distribution supplies means, medians, quantiles, prediction intervals, and tail risks from the same forecast.

Those quantities matter when errors carry asymmetric costs. Underestimating insurance losses, drug response, inventory demand, or financial risk can have different consequences from overestimating them. A mean prediction alone cannot describe the range or probability of either outcome.

Proper scoring rules evaluate probabilistic forecasts by combining accuracy with uncertainty quality. Under a strictly proper rule, a forecaster minimizes its expected penalty by reporting its actual predictive distribution. These rules are established in weather forecasting and probabilistic classification, but regression leaderboards have used them less consistently.

Inside ScoringBench

ScoringBench evaluates probabilistic tabular regression across 97 datasets from several domains. It reports conventional point metrics alongside CRPS, weighted CRPS, interval score, energy score, and CRLS. The tested models include in-context learners, fine-tuned foundation models, gradient-boosted trees, and multilayer perceptrons. Named entries include TabICL and realTabPFNv2.5 variants fine-tuned with different scoring-rule objectives.