Picking the Wrong Metric Silently Kills Classifiers in Production
A new study runs seven controlled scenarios across 15 datasets to show why single-metric reporting produces misleading conclusions about model quality.
Takeaways
- Paper reframes evaluation as a decision-oriented process across 15 UCI/OpenML datasets and seven controlled scenarios.
- Random Forest hit 0.815 accuracy on credit default data but only 0.385 MCC, exposing minority-class blindness.
- Decision Tree beat Random Forest on Car Evaluation accuracy but produced 9x worse log loss from overconfident errors.
- RMSE inflated 50-90 percent over MAE on heavy-tailed regression targets like California Housing and power consumption.
- MAPE reached 16.6 percent on Seoul Bike Sharing despite modest absolute errors, driven by near-zero denominators.
- Full paper on arXiv; datasets are public, no code repository released.
If you have ever shipped a classifier that looked stellar in the notebook and cratered in production, you are the target audience for a new arXiv paper that treats model evaluation as a first-class engineering problem rather than a reporting formality. The authors run seven controlled scenarios across 15 UCI and OpenML datasets and argue that metric disagreement is an expected, interpretable consequence of applying the wrong yardstick to non-ideal data.
When one number hides the truth
Most workflows still condense model quality into a single scalar. That works when classes are balanced, errors are symmetric, and targets are well behaved, which is rarely true in the wild. The paper focuses on four regimes where scalar reporting reliably lies: class imbalance in binary classification, asymmetric misclassification costs, multiclass prevalence skew, and heterogeneous regression error structure driven by outliers or heavy tails.
The framing is decision-theoretic. Rather than asking which metric is mathematically valid, the authors ask which one aligns with the operational cost of an error. In fraud, a missed default is real money; in oncology, a missed malignancy is a life. Those asymmetries should determine the metric, not the other way around.