Warwick and Oxford Researchers Fix a Silent Bug Crashing SciPy's Power Transforms
A new paper diagnoses why Box-Cox and Yeo-Johnson transforms silently break on ordinary data, ships a fix already merged into SciPy, and extends it to federated learning.
Takeaways
- Standard Box-Cox and Yeo-Johnson implementations silently crash on simple skewed or clustered data.
- Fix combines log-domain variance, reformulated NLL, and Lambert W bounds on lambda.
- Solution merged into SciPy, plus standalone powertf library.
- Prior 2022 NeurIPS method (ExpSearch) fails on seven real features from Cancer, Ecoli, House datasets.
- Federated variant needs only 4 numbers per client per round, converges in 20-30 rounds.
- Stable version is ~30x slower but stays well under perceptible latency.
Power transforms like Box-Cox and Yeo-Johnson are one of those quiet workhorses of ML pipelines. You throw skewed features at them, they push the distribution closer to Gaussian, and downstream models like logistic regression or LDA get happier. A new paper from researchers at Warwick and Oxford, Power Transform Revisited, shows that standard implementations of these transforms are quietly broken, and can crash or produce wildly wrong answers on datasets as trivial as [10, 10, 10, 9.9].
The authors' fix has already been merged into SciPy, and they released a companion library, powertf, that also handles the federated learning case.
The bug hiding in your preprocessing step
Box-Cox raises data to a power lambda and normalizes; Yeo-Johnson extends it to handle negative values. The parameter lambda is fit by minimizing a negative log-likelihood (NLL). Everything is convex and mathematically well-behaved, so what could go wrong?
Floating point, mostly. On simple inputs the true optimum can be very large in magnitude, causing numerical overflow when evaluating the power functions, for example 0.1 to the -361 or 10 to the 358. The authors identify three conditions that reliably push lambda into overflow territory:
- All values on one side of 1. Box-Cox has psi(lambda, 1) = 0 for all lambda, so datasets with every x greater than 1, or every x less than 1, tend to trigger positive or negative overflow respectively.