Cambridge Finds Gemma 3 27B Beats Llama 70B at Building Multilingual AI Data
A new study finds that picking the biggest model as your synthetic data teacher is often the wrong move for non-English languages.
ProTakeaways
- Cambridge researchers introduce Polyglot Score, a metric combining intrinsic data quality with student performance across 6 languages.
- They evaluated 10 teacher models, generating 1.4M SFT examples and training 240 student models on OLMo 3 7B.
- Gemma 3 27B (0.726) and Aya Expanse 32B (0.706) top the ranking, beating Llama 3.1 70B (0.140).
- Model scale does not predict teacher effectiveness; prompt diversity, length, and response fluency explain over 93% of variance.
- Matching teacher and student model families yields at least 20% higher Polyglot Score than mismatched pairs.
- Code, data, and models released at HuggingFace and GitHub.
If you have ever tried to bootstrap a multilingual dataset, you probably reached for the largest instruction-tuned model you could afford, generated a pile of prompt-response pairs, and shipped it. A new study from the University of Cambridge argues that this default quietly wrecks the quality of synthetic data for anything that isn't English, and it lays out a systematic way to pick a better teacher.
The paper introduces the Polyglot Score, a metric that combines intrinsic measures of generated data quality with the downstream performance of student models trained on it. To validate it, the authors evaluated 10 teacher models across 6 typologically diverse languages, generating over 1.4M SFT examples and finetuning 240 student models from OLMo 3 7B.
Why the biggest teacher is often the wrong teacher
Frontier models are typically benchmarked and optimized for English. When you use them to synthesize Arabic or Indonesian training data, their capability gaps in those languages leak straight into the student. Prior work tends to select teachers arbitrarily, defaulting to whichever state-of-the-art model tops English leaderboards, and the result is synthetic data that propagates the teacher's weaknesses rather than its strengths.
The Cambridge team formalizes teacher evaluation with two complementary signals. On the intrinsic side, they measure prompt and response diversity via embedding cosine distances, response perplexity under the base student, and a reward score from M-Prometheus 14B acting as a multilingual judge. On the extrinsic side, they compute the