NYU's Deletion Tuning Matches Full Fine-Tuning by Deleting 0.05% of Weights
Deletion Tuning learns a binary keep-or-delete mask over gate projections, matching full fine-tuning accuracy while zeroing under 0.05% of weights.
- Deletion Tuning matches full fine-tuning by zeroing under 0.05% of weights with no other changes (paper).
- Trained adapter is one bit per weight: a 30 MB mask for a 61 GB, 30B MoE model.
- Prior-task accuracy stays near base while FFT, LoRA, and random sparse tuning drop up to 5.4 points.
- Works across RL with verifiable rewards, supervised fine-tuning, and on-policy distillation.
- Masks only the gate projection of transformer blocks; router untouched in MoE models.
- Code, trained masks, and project page available at agenticlearning.ai/deleting-weights.
Researchers at NYU’s Agentic Learning AI Lab and the University of Calgary report that pretrained language models can acquire task-specific behavior by disabling a small set of existing weights. Deletion Tuning freezes the base checkpoint and learns a binary mask over the gate projections in transformer blocks.
The authors report that zeroing fewer than 0.05% of a model’s weights reaches accuracy comparable to full fine-tuning on the largest models they evaluated. Their experiments cover reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation across dense Llama and Qwen models and a 30B mixture-of-experts model.
Hundreds of variants, one frozen checkpoint
Common post-training methods create additional parameters for every specialized model. Full fine-tuning, or FFT, updates many continuous weight values and typically produces another full checkpoint. LoRA stores smaller low-rank matrices, but each adapter still contains newly learned values and can alter capabilities acquired during pretraining.
| Method | Learned change | Deployment artifact | Main constraint |
|---|---|---|---|
| Full fine-tuning | Continuous updates across tuned parameters | Full checkpoint or dense delta | High storage cost per variant |
| LoRA | Continuous low-rank matrices | Adapter or merged checkpoint | Update rank is capped |
| Random sparse tuning | Continuous updates at preselected positions | Sparse delta | Positions are fixed before training |
| Deletion Tuning | Binary keep-or-delete decisions | Bit mask | Target weights retain their original value or become zero |
Compact, non-destructive updates matter when one base model supports many customers, tasks, or sequential training stages. For the 30B mixture-of-experts example, the authors report a 30 MB mask associated with a 61 GB base checkpoint. Each deployment can share the frozen checkpoint and load its own mask.
How a gate becomes a bit mask
A transformer’s feed-forward network expands each token representation, applies nonlinear processing, and projects it back to the model dimension. Many current architectures use a gated feed-forward network in which a gate projection controls how strongly intermediate features pass through the block.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
