Nottingham Researchers Cut Neural Network Training Updates by 80% Using Mistake Gating

Every time a neural network sees a training sample, it updates its weights, even when the prediction was already correct. Brains do not work that way. A pair of researchers at the University of Nottingham argue this behavior is biologically odd and wasteful, and they have released a training rule that ignores samples the network already gets right, then adds a small twist to keep generalization intact.

Their paper, titled Mistake gating leads to energy and memory efficient continual learning, proposes what the authors call memorized mistake-gated learning. The rule gates synaptic updates strictly by current and past classification errors, cutting the number of updates by 50 to 80 percent. Code is on GitHub.

The cost of updating on everything

Standard backprop uses a differentiable surrogate loss like cross-entropy, so the gradient is almost never exactly zero, even on samples the network classifies correctly. Weights get nudged on every example forever. Biological learning is metabolically costly: flies subjected to aversive conditioning and then starved died 20 percent sooner than controls, and flies doubled sucrose consumption after long-term memory formation. Evolution seems to have pushed brains toward learning frugally.

Prior ML work tried to address this with core-sets, importance sampling, and example forgetting scores, but finding the core-set is hard and can require a separate model trained beforehand to identify them. None of these approaches work cleanly in an online, continual setting where you see each sample once and cannot pre-analyze the dataset.