Skip to main content
Back

Bocconi Researchers Prove SHAP Scores Impossible Data, Offer Three Fixes

A new paper shows SHAP, Breiman importance, and Knockoffs all extrapolate on correlated data, and proposes Gaussian-copula fixes with theoretical guarantees.

Pro

  • Standard permutation importance (SHAP, Breiman, Knockoffs, ALE) extrapolates on correlated features, inflating scores.
  • Authors propose GCMR and GKnock: permute in Gaussian-transformed space, then map back.
  • Under a Gaussian copula, new points are provably an independent (or knockoff) copy of the originals.
  • Restricted permutation importance is shown to equal twice the classical Sobol' total index.
  • On Boston Housing, ZN's importance drops from ~220 to ~0.72, revealing RAD and TAX as true drivers.
  • MATLAB code available on GitHub; paper was INFORMS Data Mining 2024 Best Paper runner-up.

Feature importance is one of the most-used tools in applied machine learning, yet the standard recipe of shuffling a column and measuring the loss increase has a well-documented bug. When inputs are correlated, shuffling breaks the joint distribution and forces the model to score points that could not exist in reality. A new paper from a team at Bocconi, Duke, SKEMA and Clausthal argues the problem runs deeper than commonly thought, and offers three fixes with theoretical guarantees.

When explanations score impossible data

The critique in Trustworthy Feature Importance is blunt: every non-trivial permutation-based variable importance measure, including SHAP, Knockoffs, conditional model reliance, and accumulated local effect (ALE) plots, suffers from extrapolation. A permuted dataset can contain children who are retired or people who graduated from high school before they were born, and asking the model to predict on those points produces nonsense that then gets aggregated into the importance score.

The paper makes this concrete on Boston Housing. When SHAP is computed for a linear regression fit to the data, house 63 receives a SHAP value of 448 for TAX despite a market value of 8.8 and a predicted value of 6.0. Some of the intermediate predictions used to compute that value drop to around minus 1000, so the explanation is being built from prices the model would never actually output on real data.

Three fixes, one common trick