VISTEC's DuDi Teaches Tiny AI Models to Speak Southeast Asian Languages
DuDi combines three distillation signals and a cross-lingual verbalizer to dramatically improve sub-billion parameter models on Southeast Asian languages
- DuDi is a new distillation framework that teaches sub-billion parameter models to handle Southeast Asian languages by combining three complementary training signals.
- The key innovation is a cross-lingual verbalizer that routes teacher outputs through a high-resource language for reasoning, then produces answers in the target language, maximizing teacher-student alignment.
- Evaluated on SEA-HELM, DuDi consistently outperforms SFT, SeqKD, and GKD baselines across multiple model families and scales.
- The primary test setup distills Qwen2.5-3B-Instruct into Qwen2.5-0.5B, a 6x compression ratio, across Filipino, Indonesian, Tamil, Thai, and Vietnamese.
- A cold-start SFT initialization on ground-truth data is required for best performance -- skipping it significantly hurts results.
- Code and model checkpoints are linked from the paper page on GitHub and Hugging Face.
Getting a large language model to speak Thai, Vietnamese, or Filipino fluently is hard enough. Getting a tiny model -- one small enough to run on a phone or an edge device -- to do the same is a problem that has largely gone unsolved. A new paper from researchers at VISTEC, AI Singapore, and MBZUAI introduces DuDi, a distillation framework specifically designed to close that gap for Southeast Asian languages.
The wall that small models hit
Small language models are efficient and scalable, but their multilingual capabilities degrade severely at sub-billion scales, especially for Southeast Asian languages. This is not a minor performance dip -- it is a cliff. Models below 1B parameters tend to collapse into English-dominant behavior, producing responses in the wrong language or losing grammatical coherence entirely when prompted in low-resource languages.
The standard fix is knowledge distillation (KD): train a small "student" model to mimic the outputs of a larger, more capable "teacher" model. KD transfers knowledge from a large, pre-trained teacher model to a smaller, more efficient student model by aligning outputs or intermediate representations. But while effective for moderate-scale teacher models, traditional KD struggles with LLMs due to their vast scale, where knowledge is distributed across billions of parameters and intricate attention patterns. In the multilingual case, there is an additional wrinkle: the teacher and student often respond in different languages to the same prompt, making it nearly impossible to align their output distributions meaningfully.
Why existing distillation breaks in multilingual settings
Most distillation approaches fall into two camps. Off-policy methods train the student on sequences the teacher already generated -- fast and stable, but the student never learns to handle its own mistakes. On-policy methods let the student generate its own outputs and then score them against the teacher -- more realistic, but prone to distribution collapse when teacher and student diverge too much.
In multilingual settings, this divergence is especially severe. A teacher model might respond to a Thai question in Thai, while the student -- undertrained on Thai -- responds in English. When you try to compute a token-level loss between those two sequences, the signal is essentially noise: the tokens don't correspond to each other at all. DuDi scores student rollouts with a teacher routed through a cross-lingual verbalizer, repairing teacher-student transfer for multilingual students.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
