Sheffield's SSU Teaches LLMs Rare Languages Without Breaking Them
A new selective freezing method preserves an instruct model's English abilities while teaching it low-resource languages, cutting forgetting from 20% to 3%.
Takeaways
- SSU freezes parameters critical to source-language abilities before adapting an LLM to a new language.
- Source-language forgetting drops from ~20% (full fine-tuning) to ~3% while matching target-language gains.
- Uses Wanda-style importance scoring on 500 source samples, then freezes top 50% of columns per weight matrix.
- Tested on OLMo 2 7B/13B across Nepali, Kyrgyz, Amharic, Hausa, and Igbo.
- Reduces code-mixing in responses to 1% vs 6-17% for other selective-update baselines.
- Code and models released at github.com/gucci-j/ssu with negligible training overhead.
Teaching an instruct-tuned LLM a new language is one of the fastest ways to break it. Run a round of continual pre-training on Nepali or Amharic text, and the model often loses much of what made it useful in the first place: instruction-following, safety alignment, chat coherence, and reasoning. A team from the University of Sheffield, Hitachi, and the University of Exeter has proposed a fix called Source-Shielded Updates (SSU), and the numbers are striking.
SSU trims performance degradation on monolingual source tasks to 3.4% (7B) and 2.8% (13B) on average, against 20.3% and 22.3% from full fine-tuning. It also matches or beats full fine-tuning on the target-language benchmarks it was trying to learn in the first place.
Why adapting instruct models is uniquely painful
The standard recipe for adding a language to an LLM is continual pre-training on target-language text. That works reasonably well for base models, but instruct models are more fragile. They require specialized instruction-tuning data, which is often unavailable or prohibitively costly to produce for underrepresented languages, and machine-translated instruction data tends to be unreliable. So the only realistic option for most low-resource languages is raw unlabeled text.
Raw text carries no chat templates, no user/assistant roles, no instruction structure. Optimizing an instruct model on it drags the weights away from the format the model was aligned to. Prior work tried to patch this up after the fact through weight merging or task-vector arithmetic, but those methods largely fail to mitigate catastrophic forgetting, substantially degrading core functionalities.