Loughborough Stops AI From Forgetting Everything After 30 Data Deletions
A new SVD-guided training constraint keeps LoRA adapters from colliding across dozens of unlearning requests, avoiding costly router overhead at inference.
Takeaways
- New method uses SVD to constrain each LoRA unlearning update to an orthogonal subspace, preventing parameter collision.
- After 30 sequential forgets on CIFAR-100, retained accuracy holds at 58.06% versus 12.74% for static fusion.
- Eliminates MoE routing overhead: no adapter registry, no per-inference router, just standard fused weights.
- Two variants tested: post-hoc projection and in-training constrained optimization, with in-training performing best.
- Hard ceiling on task count from finite subspace budget; saturates around 15-16 tasks on a 64-dim head.
- Paper on arXiv; evaluated on ResNet-20/CIFAR-100 and MLP/MNIST, no public code yet.
Every time a user invokes their right to be forgotten, a deployed model is supposed to erase what it learned from their data without wiping out everything else. Doing that once is manageable. It becomes brittle when deletion requests keep arriving, and each one leaves a new patch stitched into the same weights. A team from Loughborough University has proposed a way to keep those patches from stepping on each other, using singular value decomposition to carve out orthogonal subspaces for each new LoRA update.
Their paper reports that after thirty sequential class-forgetting tasks on CIFAR-100, standard static LoRA fusion drops retained accuracy from 60.39% down to 12.74%, while their in-training constrained version holds at 58.06%. That is the difference between a model that still works and one that has effectively been lobotomized.
Why sequential forgetting breaks LoRA
LoRA is the natural tool for cheap unlearning: instead of retraining, you learn a small low-rank update that cancels the influence of the forget set, then fuse it back into the base weights. The trouble is that when many such updates are learned sequentially and then fused into a single model, their effects can overlap in the same parameter directions. That overlap, called parameter collision, looks a lot like catastrophic forgetting in continual learning.
The current workaround is dynamic routing. Methods like AC-LoRA and I-LoRA rely on Mixture-of-Experts routing, where a small network decides which adapter should be emphasized for each input. It works, but you pay for it at inference: all adapters must remain available, extra routing parameters must be stored, and each forward pass requires a routing decision before combining the adapter outputs. For thirty forgetting tasks that means thirty adapters loaded in memory plus a router evaluated on every forward pass.