HAE-RAE's KMMMU Benchmark Exposes Gemini-3-Pro Struggling at 52% on Korean Exams
A new Korean-native benchmark of 3,466 exam questions exposes how top vision-language models stumble on local conventions and expert visual formats.
ProTakeaways
- KMMMU is a 3,466-question native Korean multimodal benchmark spanning nine disciplines and nine visual modality types, available on Hugging Face
- Best open-source model (Qwen3-VL-235B-A22B-Thinking) reaches only 42.05% on the full set; Gemini-3-Pro tops the 627-item hard subset at 52.42%
- Korean-specific questions cause up to a 13.43 percentage point drop for strong multilingual models, even after controlling for discipline mix
- Questions come from four Korean exam pipelines: PSAT, National Technical Qualifications, National Competency Standards, and academic Olympiads
- Adversarial filtering removed items solvable by Phi-3.5-Vision, InternVL-3.5-38B, and two Gemini-2.5-Flash variants, cutting 68k candidates to 3,466
- Error analysis points to convention-to-label mismatches, weak few-shot rule induction, and shallow localized knowledge rather than reasoning depth
A research team from Chung-Ang University, Seoul National University, and HAE-RAE has released KMMMU, a Korean-native multimodal benchmark that puts frontier vision-language models through the same exams Korean professionals take. The results are unflattering: Gemini-3-Pro tops out at 52.42% on the hard subset, and the strongest open-source contender manages only 42.05% on the full set.
The benchmark targets something existing evaluations largely ignore: whether multimodal models can handle information-dense, institutionally grounded content in a non-English language. Prior work leans English-centric or relies on translated datasets, which fail to capture tasks shaped by local institutional conventions, discipline-specific formats, and dense visual materials in non-English contexts.
Inside the question bank
KMMMU contains 3,466 questions drawn from Korean assessment sources, spanning nine disciplines, nine visual modality categories, and both multiple-choice and open-form formats. The team pulled from four high-stakes exam pipelines in South Korea:
- PSAT: ten years of civil service recruitment exams covering logical reasoning and information integration
- National Technical Qualifications: fifteen years of items from 252 certification exams (Information Processing Engineer, Electric Engineer, Fire Safety Manager, and others)
- Olympiads: ten years of math and science problems spanning middle school through university
- NCS: three years of National Competency Standards exams used in public sector hiring