---
title: "VISTEC's DuDi Teaches Tiny AI Models to Speak Southeast Asian Languages"
subtitle: "DuDi combines three distillation signals and a cross-lingual verbalizer to dramatically improve sub-billion parameter models on Southeast Asian languages"
url: "https://alphasignal.ai/news/vistec-s-dudi-teaches-tiny-ai-models-to-speak-southeast-asian-languages"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-03T00:00:00.000Z"
updated: "2026-06-12T01:01:29.045Z"
type: "Paper"
topics:
  - "Post Training"
  - "Llms"
subtopics:
  - "Distillation"
  - "Small Models"
  - "Translation"
upvotes: 3468
source_url: "https://arxiv.org/abs/2606.04694"
comments: 0
truncated: true
---

# VISTEC's DuDi Teaches Tiny AI Models to Speak Southeast Asian Languages

> DuDi combines three distillation signals and a cross-lingual verbalizer to dramatically improve sub-billion parameter models on Southeast Asian languages

Getting a large language model to speak Thai, Vietnamese, or Filipino fluently is hard enough. Getting a _tiny_ model -- one small enough to run on a phone or an edge device -- to do the same is a problem that has largely gone unsolved. A new paper from researchers at VISTEC, AI Singapore, and MBZUAI introduces [DuDi](https://arxiv.org/abs/2606.04694), a distillation framework specifically designed to close that gap for Southeast Asian languages.

## The wall that small models hit

Small language models are efficient and scalable, but their multilingual capabilities degrade severely at sub-billion scales, especially for Southeast Asian languages. This is not a minor performance dip -- it is a cliff. Models below 1B parameters tend to collapse into English-dominant behavior, producing responses in the wrong language or losing grammatical coherence entirely when prompted in low-resource languages.

The standard fix is **knowledge distillation** (KD): train a small "student" model to mimic the outputs of a larger, more capable "teacher" model. KD transfers knowledge from a large, pre-trained teacher model to a smaller, more efficient student model by aligning outputs or intermediate representations. But while effective for moderate-scale teacher models, traditional KD struggles with LLMs due to their vast scale, where knowledge is distributed across billions of parameters and intricate attention patterns. In the multilingual case, there is an additional wrinkle: the teacher and student often respond in different languages to the same prompt, making it nearly impossible to align their output distributions meaningfully.

## Why existing distillation breaks in multilingual settings

Most distillation approaches fall into two camps. **Off-policy** methods train the student on sequences the teacher already generated -- fast and stable, but the student never learns to handle its own mistakes. **On-policy** methods let the student generate its own outputs and then score them against the teacher -- more realistic, but prone to distribution collapse when teacher and student diverge too much.

In multilingual settings, this divergence is especially severe. A teacher model might respond to a Thai question in Thai, while the student -- undertrained on Thai -- responds in English. When you try to compute a token-level loss between those two sequences, the signal is essentially noise: the tokens don't correspond to each other at all. DuDi scores student rollouts with a teacher routed through a cross-lingual verbalizer, repairing teacher-student transfer for multilingual students.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer](https://arxiv.org/abs/2606.04694)
- [SEA-HELM: Southeast Asian Holistic Evaluation of Language Models](https://arxiv.org/abs/2502.14301)
- [SEA-HELM Documentation](https://docs.sea-lion.ai/benchmarks/sea-helm)
- [Awesome LLM On-Policy Distillation (GitHub)](https://github.com/nick7nlp/Awesome-LLM-On-Policy-Distillation)
- [DuDi HTML paper (arXiv)](https://arxiv.org/html/2606.04694)