Researchers Expose How Equally Good Models Tell Wildly Different AI Stories
A new metamorphic testing framework exposes when SHAP and LIME explanations are untrustworthy — without needing ground-truth labels
- Core idea: A new framework uses metamorphic testing to check if SHAP/LIME explanations are faithful — no ground-truth labels needed.
- The problem: Multiple models with equal accuracy can produce wildly different feature explanations (the Rashomon effect), making it unclear which to trust.
- How it works: Five logical consistency rules (metamorphic relations) are applied across a pool of near-equivalent models (the epsilon-Rashomon set) to score explanation reliability.
- Key finding: LIME produces suspiciously homogeneous explanations across models, while SHAP shows more variation — ~14% of model pairs violate cross-model sensitivity consistency.
- Best use-case: Model selection in regulated domains (finance, healthcare) where explanation quality is a compliance requirement, not just a diagnostic tool.
- Limitation: Evaluated only on two tabular regression datasets; no public code released; scaling to deep learning remains unaddressed. Read the paper.
You trained five models. They all hit the same validation score. You pick one, run SHAP on it, and present the feature importances to your stakeholders. But here's the uncomfortable truth: had you picked a different model from that equally-good pool, SHAP might have told a completely different story. This is the Rashomon effect in explainable AI , and a new paper from Helge Spieker, Jørn Eirik Betten, and Arnaud Gotlieb proposes a principled way to catch it before it erodes trust in your system.
The problem no one talks about at model selection time
The Rashomon effect in machine learning describes the situation in which many models fit the training data equally well but tell different stories about which features matter. This isn't a niche edge case , it's the default state of modern ML pipelines, where hyperparameter sweeps and ensembles routinely produce dozens of near-equivalent candidates. When such models are paired with post-hoc explanation methods, stakeholders , end users, regulators, domain experts , may receive conflicting accounts of the same phenomenon depending on which model is selected, undermining trust and actionability.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
