Skip to main content
Back

UCL's iSDFT Keeps AI Sharp on New Tasks Without Forgetting Old Skills

A new fine-tuning method treats the teacher model as a budgeted information source, letting LLMs pick up new skills without erasing old ones.

Paper, Full Access
  • UCL and Huawei Noah's Ark researchers propose iSDFT, a per-token information-budgeted distillation method.
  • Extends prior SDFT work that used the model as its own on-policy teacher.
  • Adds a frozen base-policy anchor to prevent cumulative drift during long training runs.
  • Beats vanilla SDFT in 7 of 8 model-task settings across four LLM backbones.
  • 73% of retention evaluations stay within 0.5 points of the base model, versus 52% for the strongest baseline.
  • Largest mean improvement on ten math, coding, and competition-math benchmarks tested.

Fine-tuning a large language model for a narrow task can degrade capabilities learned during pretraining. Researchers at University College London and Huawei Noah's Ark Lab propose iSDFT, an extension of self-distillation that controls how much information a teacher contributes at each prediction state. The method also anchors training to the frozen base model, aiming to improve specialization without allowing unrelated skills to drift as far.

Why narrow tuning erases broad skills

A narrow training set updates parameters shared across many behaviors. Those updates can improve the target task while changing token distributions used for reasoning, coding, factual recall, and instruction following. This degradation is commonly called catastrophic forgetting.

Organizations often contain the problem with separate fine-tuned models, task-specific adapters, or repeated regression testing. Those approaches preserve isolation but add serving costs, versioning work, and governance overhead when tasks accumulate.

SDFT still points at the full teacher

Self-Distillation Fine-Tuning generates an on-policy rollout from the student, then asks the same underlying model, conditioned on a demonstration, to produce a teacher distribution for each generated prefix. The student learns from those distributions instead of relying only on static supervised targets. Earlier work from MIT, the Improbable AI Lab, and ETH Zurich reported better retention with this approach; additional context appears in the SDFT background.

Vanilla SDFT always uses the full demonstration-conditioned teacher as its target. The distance between student and teacher still varies across tokens, so update magnitudes are not literally identical. However, the method provides no explicit way to specify how much teacher information should transfer at a given prediction state.