---
title: "UCL's iSDFT Keeps AI Sharp on New Tasks Without Forgetting Old Skills"
subtitle: "A new fine-tuning method treats the teacher model as a budgeted information source, letting LLMs pick up new skills without erasing old ones."
url: "https://alphasignal.ai/news/ucl-s-isdft-keeps-ai-sharp-on-new-tasks-without-forgetting-old-skills"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-09-21T14:21:50.000Z"
updated: "2026-09-26T01:58:23.226Z"
type: "Paper"
topics:
  - "Post Training"
  - "Llms"
subtopics:
  - "Distillation"
  - "Fine Tuning"
  - "Hallucinations"
upvotes: 10
source_url: "https://arxiv.org/abs/2609.24646"
comments: 0
truncated: true
---

# UCL's iSDFT Keeps AI Sharp on New Tasks Without Forgetting Old Skills

> A new fine-tuning method treats the teacher model as a budgeted information source, letting LLMs pick up new skills without erasing old ones.

Fine-tuning a large language model for a narrow task can degrade capabilities learned during pretraining. Researchers at University College London and Huawei Noah's Ark Lab propose [iSDFT](https://arxiv.org/abs/2609.24646), an extension of self-distillation that controls how much information a teacher contributes at each prediction state. The method also anchors training to the frozen base model, aiming to improve specialization without allowing unrelated skills to drift as far.

## Why narrow tuning erases broad skills

A narrow training set updates parameters shared across many behaviors. Those updates can improve the target task while changing token distributions used for reasoning, coding, factual recall, and instruction following. This degradation is commonly called catastrophic forgetting.

Organizations often contain the problem with separate fine-tuned models, task-specific adapters, or repeated regression testing. Those approaches preserve isolation but add serving costs, versioning work, and governance overhead when tasks accumulate.

## SDFT still points at the full teacher

Self-Distillation Fine-Tuning generates an on-policy rollout from the student, then asks the same underlying model, conditioned on a demonstration, to produce a teacher distribution for each generated prefix. The student learns from those distributions instead of relying only on static supervised targets. Earlier work from MIT, the Improbable AI Lab, and ETH Zurich reported better retention with this approach; additional context appears in the [SDFT background](https://www.infoworld.com/article/4131242/researchers-propose-a-self-distillation-fix-for-catastrophic-forgetting-in-llms.html).

Vanilla SDFT always uses the full demonstration-conditioned teacher as its target. The distance between student and teacher still varies across tokens, so update magnitudes are not literally identical. However, the method provides no explicit way to specify how much teacher information should transfer at a given prediction state.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs](https://arxiv.org/abs/2609.24646)
- [iSDFT full HTML paper](https://arxiv.org/html/2609.24646v1)
- [Researchers propose a self-distillation fix for catastrophic forgetting in LLMs](https://www.infoworld.com/article/4131242/researchers-propose-a-self-distillation-fix-for-catastrophic-forgetting-in-llms.html)
- [Self-Distillation Fine-Tuning (SDFT) overview](https://www.emergentmind.com/topics/self-distillation-fine-tuning-sdft)
