FFR Brings Geoffrey Hinton's Forward-Forward Algorithm to Regression at 8% Memory
FFR extends Hinton's Forward-Forward algorithm to regression tasks, cutting training memory to just 8% of backpropagation's at depth 32 while recovering 98.6% of BP accuracy
Takeaways
- First FF regression framework: FFR extends Hinton's Forward-Forward algorithm to continuous regression tasks for the first time at real-world scale.
- Massive memory savings: Peak training memory drops to 27% of backpropagation's at depth 8, and just 8% at depth 32.
- Near-BP accuracy: FFR recovers 98.6% of BP's accuracy on average across five real-world regression benchmarks.
- Three core innovations: Ordinal competitive goodness function, stratified ladder architecture, and hierarchical prediction with free uncertainty estimation.
- Faster per-iteration: Per-iteration compute runs at ~72% of BP's wall-clock time while outperforming all other BP-free methods.
- Read the paper: arxiv.org/abs/2606.03927 -- no public code repository linked yet.
Geoffrey Hinton's Forward-Forward (FF) algorithm has been one of the most intriguing ideas in recent deep learning research: replace the backward pass entirely with a second forward pass, letting each layer learn from purely local signals. No global gradient. No storing intermediate activations. The memory savings are dramatic, and the biological plausibility argument is compelling. There is one catch -- it only works for classification. Until now.
A new paper, FFR: Forward-Forward Learning for Regression, claims to be the first framework to successfully extend FF to real-world regression tasks, and the numbers are hard to ignore: 98.6% of backpropagation's accuracy on average, peak training memory reduced to just 8% of BP's at depth 32, and per-iteration compute time at roughly 72% of BP's.
Why regression breaks the Forward-Forward algorithm
To understand why this is hard, you need to understand how FF works. The algorithm replaces the traditional forward and backward passes of backpropagation with two forward passes: one using positive (real) data and another using negative data. Each layer in the neural network is associated with its own objective function to maximize the "goodness" for positive data and minimize it for negative data. The "goodness" is typically the sum of squared activations -- a scalar score that tells each layer whether the input it received looks real or fake.
This works beautifully for classification because you can construct a clear negative sample: just slap the wrong label onto an input. The network learns to produce high goodness for correct label-input pairs and low goodness for incorrect ones. In the inference phase, the model evaluates all possible one-hot encodings on an unseen input. Since the network has been trained to produce high goodness values only for correct labels, the encoding that results in the highest total goodness across all layers is selected as the predicted class.
Regression destroys this setup. When your target is a continuous number -- say, predicting temperature, tool wear, or indoor location -- there is no natural "wrong" label to construct a negative sample from. A target of 23.4 does not have an obvious opposite. And even if you tried to construct one, the standard goodness function (sum of squared activations) carries no information about whether a prediction is close to or far from the true value. The layer has no way to know whether it is warm or cold.