Skip to main content
Back

aiXplain Fixes Hinton's Forward-Forward Algorithm, Closing the Gap to Backprop by 32%

Rethinking the goodness function turns Hinton's Forward-Forward algorithm from a curiosity into something that nearly matches backprop on MNIST.

Pro

  • aiXplain researchers redesign the Forward-Forward goodness function, gaining +32.6pp on Fashion-MNIST over the sum-of-squares baseline.
  • Top-k goodness measures only the ~2% most active neurons, jumping Fashion-MNIST accuracy from 56.4% to 79.0%.
  • Burstiness (excess kurtosis) goodness is parameter-free, scale-invariant, and reaches 88.1% on Fashion-MNIST with negligible overhead.
  • FFCL + burstiness hits 98.18% on MNIST, effectively matching backpropagation's ~98.4% ceiling for this architecture.
  • Three sweeps (k, alpha, moment order) all trace inverted-U curves, confirming sparsity is the key design choice.
  • Code and configs at the paper's anonymous repo reproduce the full suite in ~4 A100 GPU-hours.

The Forward-Forward algorithm, introduced by Geoffrey Hinton in 2022, promised a biologically plausible alternative to backpropagation. Instead of a global backward pass, each layer trains itself locally by pushing up a scalar 'goodness' score for real data and pushing it down for fake data. The catch: it has always trailed backprop by a wide margin, and everyone kept using the same goodness function Hinton picked as a placeholder. A new paper from aiXplain argues that this one function was the bottleneck all along.

The one line nobody questioned

In vanilla Forward-Forward, each layer's goodness is the sum of squared activations (SoS). Positive samples (image plus correct one-hot label) should produce large SoS, negative samples (image plus wrong label) should produce small SoS. This choice has remained essentially unquestioned across follow-up work on layer sizes, negative data, layer collaboration, and predictive coding variants.

The authors argue that SoS is wrong in principle. By focusing exclusively on sum-of-squares, prior work has implicitly assumed that all neurons contribute equally to the learning signal, or that magnitude alone is sufficient for discrimination. Their unifying principle is that the goodness function must be sensitive to the shape of neural activity rather than its total energy.

Two families of fixes