Meta and Technion Crack the Optimal Regularization Formula for Continual AI Learning
New theory pins down the optimal L2 regularization strength for sequential training, showing it scales nearly linearly with the number of tasks.
ProTakeaways
- New paper derives closed-form generalization loss for L2-regularized continual linear regression in high dimensions.
- Optimal regularization strength λ* scales as T/ln T with the number of tasks.
- Practical recipe: tune λ on a short horizon, then multiply linearly for long horizons.
- Result is the first characterization of optimal regularization strength for multi-teacher continual learning.
- Any fixed λ eventually hurts as tasks accumulate under teacher variance.
- T/ln T scaling holds empirically for MNIST with a two-layer ReLU network too.
Continual learning is easy to describe and painful to tune. You train a model on task 1, then task 2, then task 3, and by task 50 you either have a model that remembers nothing from the start or one so locked in place it can't absorb anything new. A group from Technion, Edinburgh, and Meta just published a clean theoretical result that removes at least one dimension of that tuning nightmare: they derive the optimal L2 regularization strength for continual linear regression and show it grows almost linearly with the number of tasks.
The tuning tax nobody wants to pay
The workhorse trick for fighting catastrophic forgetting is to add a penalty that keeps new weights close to the previous ones. At each iteration t, the learner updates the predictor by minimizing squared error on the current task plus a term λd times the squared distance from the previous iterate. The scalar λ controls the classic stability versus plasticity trade off: too small and old knowledge gets overwritten, too large and the model refuses to learn anything new.
In practice, engineers grid search λ. That works for two or three tasks. For a foundation model updated across hundreds of task shards, sweeping λ end to end becomes absurd. A theoretically guided scaling law for the regularization strength cuts computational cost and removes most of the hyperparameter tuning burden. That is the gap this paper fills.
The setting, stripped down
The authors analyze continual linear regression in the high dimensional regime, with sample count n and dimension d both going to infinity at a fixed ratio n/d = α ≤ 1. Each task has its own "teacher" vector generating noisy labels, and the teachers themselves can vary. Allowing multiple teachers models practical scenarios where the underlying ground truth, such as user preferences or fraud behaviors, evolves over time.