Oxford Fixes Active Learning's Blind Spot by Targeting Your Actual Loss
A new Bayesian active learning framework lets data acquisition target the exact loss function you care about, not just log-likelihood or squared error.
ProTakeaways
- New framework derives active learning objectives directly from any downstream loss, not just log-likelihood or squared error
- Works analytically for any weighted Bregman divergence loss, including NLL, squared error, Box-Cox, and Linex
- Introduces weighted EPIG (classification) and weighted EVR (regression) as concrete drop-in objectives
- Beats loss-mismatched baselines by up to an order of magnitude on Linex loss experiments
- Prediction-space weighting cannot be replicated by simply retraining with a weighted loss
- Code available at github.com/Zhuoyue-Huang/loss-driven-bayesian-active-learning
Active learning is supposed to pick the most useful next data point to label, but there is a strange mismatch at the heart of how it usually works. Standard objectives quietly optimise for whichever loss happens to be mathematically convenient, not the loss you actually care about at deployment. A new paper from Oxford, Loss-Driven Bayesian Active Learning, closes that gap by deriving a principled acquisition rule directly from your downstream loss function, and it works for any objective shaped like a weighted Bregman divergence.
The convenient-loss trap
Most popular active learning methods lock you into a fixed objective. Information-theoretic scores like BALD target log loss, Bayesian variance-based scores target quadratic loss, and error-based alternatives allow more flexibility but restrict which models you can practically use. If your real-world evaluation metric happens to be one of those, fine. It usually is not.
Consider a few scenarios the authors flag. In protein design you might care more about accurately predicting a desirable property when it takes a high value. In safety-critical settings you might penalise optimistic predictions more heavily than pessimistic ones. Standard BALD does not know or care about any of that. It just reduces entropy over model parameters.
Working backwards from the decision
The framework starts from Bayesian decision theory. Assume you have a terminal loss function that scores your final action against the true world state. The Bayes-optimal action then minimises expected loss under your posterior, and the associated minimum is called the