Beijing Academy of AI's LaoBench Exposes Massive Gaps in Multilingual AI
A new 17,000-sample benchmark exposes how much top LLMs still struggle with Lao culture, curriculum, and translation compared to native speakers.
Takeaways
- LaoBench introduces the first large-scale multidimensional Lao benchmark with 17,000+ expert-curated items.
- Three subsets: Lao-7k open MCQ, Lao-10k held-out black-box MCQ, and Lao-500 open-ended prompts.
- Covers Knowledge Application, K12 curriculum, and Lao-Chinese-English translation dimensions.
- GPT-5-High leads MCQ tasks; Gemini-2.5-Pro tops translation BLEU at only mid-30s.
- Human experts hit 97%+ accuracy, revealing a persistent gap for even frontier models.
- Open subsets released at huggingface.co/datasets/BAAI/LaoBench.
Lao has roughly the same number of speakers as Swedish, yet barely registers in mainstream LLM evaluation. A team from the Beijing Academy of AI and China-ASEAN Information Harbor wants to change that with LaoBench, a 17,000-sample suite that stress-tests how well modern models understand the Lao language, K12 curriculum content, and translation between Lao, Chinese, and English.
Even the strongest closed models still trail human experts by wide margins, particularly on culturally grounded reasoning and translation fidelity. GPT-5-High tops the multiple-choice categories, Gemini-2.5-Pro leads translation BLEU, and open-source models trail noticeably, especially at smaller scale.
Why Lao breaks standard evaluation
Lao is officially spoken by millions but remains low-resource in NLP because of limited digitized corpora and scarce labeled data. The language itself throws several curveballs at LLMs:
- Ambiguous tokenization under scriptio continua (text written without spaces between words)
- A complex tonal system
- Frequent Pali and Sanskrit loanwords
Scriptio continua also complicates evaluation itself. Because Lao is written without explicit word boundaries, BLEU is sensitive to segmentation choices. The authors run Lao-aware tokenization through LaoNLP before scoring and report chrF++ as a secondary character-level metric.