Tübingen AI Center's LiT Benchmark Exposes How Multilingual AI Tests Measure English Math
A new study argues popular multilingual benchmarks quietly measure math and trivia, and proposes round-trip translation as a cleaner fix.
PRO- Multilingual benchmarks like MT-AIME24 and INCLUDE mostly measure English reasoning and factual recall, not language ability.
- Thinking variants dominate these benchmarks but do no better on LMArena human preference ratings.
- The LiT benchmark uses round-trip translation through 4-language chains scored by MQM.
- LiT scores correlate with LMArena at rho = 0.94, requiring no human references or stronger judge model.
- Frontier models collapse on low-resource languages; several score near zero on Africa and South America chains.
- Gemini-3-Flash leads at 86.7 global MQM>=80, over 10 points ahead of the next competitor.
A new paper from the Tübingen AI Center makes a claim that should unsettle anyone who trusts multilingual leaderboards: the benchmarks frontier labs use to prove multilingual progress are not really testing multilingual ability. They test mathematical reasoning and factual recall, wrapped in translated prompts. To fix this, the authors revive a decades-old technique, round-trip translation, and package it into a new benchmark called Lost in Translation (LiT).
The evidence for the mismatch is uncomfortably clean. Popular multilingual evals measure math and trivia rather than language proficiency, and thinking variants dominate on those tests while often losing to instruct variants on real-world tasks such as LMArena. Turning on chain-of-thought inflates scores on the benchmark but does not help users who actually chat with the model in their own language.
Benchmarks pointed at the wrong target
Frontier model reports lean heavily on two families of multilingual evals: multilingual math (MT-AIME24, MGSM, PolyMath) and translated general-knowledge multiple choice (MMMLU, Global-MMLU, INCLUDE). The authors correlate scores on these against LMArena Elo, the crowd-sourced human preference leaderboard, across six frontier open-source models in both Thinking and Non-Thinking variants.
The result is a near-zero or slightly negative Spearman correlation. MT-AIME24 tracks English AIME25 with rho = 0.94. INCLUDE tracks English MMLU-Pro with rho = 0.83. A model that improves at English math or English trivia will look like it got better at Swahili, even if its Swahili generation is unchanged.
Manual error analysis on Qwen3-235B-A22B-Thinking sharpens the point. On MT-AIME24 in French, Russian, and Thai, 100% of errors are logical or arithmetic, not comprehension failures. On INCLUDE in Bengali and Chinese, over 96% of errors are factual knowledge gaps. Across models, reasoning traces are almost always in English regardless of the input language. The model parses the translated problem fine. It just cannot solve it, in English, in its head.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
