George Washington University Gives CNNs Statistically Guaranteed Uncertainty Scores
A bootstrap method built on convexified CNNs delivers prediction intervals with theoretical consistency guarantees, tighter confidence bounds, and lower compute than ensembles.
ProTakeaways
- Bootstrap framework for CNN uncertainty quantification with proven asymptotic consistency, published at AAAI.
- Built on Convexified CNNs, where nuclear-norm regularization makes training a convex problem with global optima.
- Warm-starting each bootstrap from the previous solution cuts training iterations by roughly an order of magnitude.
- Transfer learning through VGG16 or ResNet50 features extends the two-layer CCNN to arbitrary deep architectures.
- Beats deep ensembles and bootstrap CNNs on log-likelihood and interval length across MNIST, Fashion MNIST, CIFAR10, and Cats and Dogs.
- Train and Forget pretraining trick was the strongest of three methods for producing independent feature extractors.
Uncertainty quantification in deep learning has always felt like a bit of a hack. You train a network, get a softmax score, and hope the number reflects real confidence. In domains like medical imaging, that guess-and-check approach falls apart quickly. A team from George Washington University and Amazon has proposed a bootstrap-based framework that puts prediction intervals for CNNs on firm statistical footing, using a convex reformulation of convolutional networks as its foundation.
Why softmax scores lie
Among the few existing uncertainty quantification approaches for deep learning, none has theoretical consistency that can guarantee the quality of the estimate. Bayesian neural networks, MC dropout, and deep ensembles all produce numbers, but there is no proof those numbers converge to the true sampling distribution as data grows.
The culprit is non-convexity. Training a standard CNN with SGD lands you at some local minimum, and which minimum depends on initialization and batch order. Applying warm-starting to a non-convex CNN drops the solution into whichever local optimum sits closest to the previous run, so predictions from the last bootstrap implicitly rely on samples not included in its training data. That cancels the statistical validity of the procedure, and bootstrap resampling, the classical tool for estimating uncertainty, quietly breaks.
The convex trick
The authors build on Convexified Convolutional Neural Networks (CCNN), a two-hidden-layer reformulation where training becomes a convex optimization problem. Instead of learning convolution filters directly, CCNN parameterizes the model by a matrix