Skip to main content
Back

GlotOCR Bench Reveals GPT-4.1 Fails on 92% of the World's Scripts

A new benchmark tests 14 OCR models across 158 Unicode scripts — and even the best frontier model fails on 92% of low-resource scripts

Pro

  • GlotOCR Bench is the first OCR benchmark covering 158 Unicode scripts, testing 14 models including GPT-4.1 and Gemini 3.1 Flash-Lite. [Paper]
  • Even the best model (Gemini 3.1 Flash-Lite) fails on 92% of low-resource script sentences, scoring only 7.7% Acc@5 on 148 low-resource scripts.
  • Performance drops sharply — not gradually — between mid-resource and low-resource scripts, suggesting a hard training coverage threshold.
  • Cross-script hallucination is the dominant failure mode: 68.4% of predictions are in the wrong script, with models defaulting to Latin, Arabic, or Devanagari.
  • Giving models explicit hints about which characters are present barely helps (+0.7pp), confirming the bottleneck is visual recognition, not script identification.
  • Benchmark and pipeline are publicly available: HuggingFace dataset and GitHub pipeline (Apache 2.0).

OCR has quietly become one of the killer apps of vision-language models. Drop a scanned document into GPT-4.1 or Gemini and it will read it back to you with impressive accuracy. But there's a catch: that impressive accuracy only holds for a small club of writing systems. A new benchmark from researchers at LMU Munich, TU Munich, and Sorbonne Université just measured exactly how small that club is , and the results are stark.

GlotOCR Bench is the first benchmark to evaluate OCR generalization across the full breadth of Unicode, covering 158 scripts. After testing 14 models , including GPT-4.1, Gemini 3.1 Flash-Lite, olmOCR-2, RolmOCR, and a range of specialized open-weight OCR systems , the conclusion is blunt: most models handle fewer than ten scripts well, and even the strongest frontier models fail to generalize beyond thirty.

A problem hiding in plain sight

The Unicode Standard currently encodes 172 scripts, representing thousands of years of human writing across every inhabited continent. Many of these scripts are still in active use by millions of speakers; others are of critical importance to historical linguistics, archaeology, and cultural preservation. The digitization of documents in these scripts depends on OCR, yet until now there was no systematic picture of where current models succeed and where they fail entirely.

These advancements have largely concentrated on a small subset of the world's writing systems. Existing benchmarks primarily focus on high-resource scripts like Latin or Chinese, leaving the vast majority of human writing systems , many of which are critical for cultural preservation and historical research , underexamined. Benchmarks like OCRBench, CC-OCR, and OmniDocBench evaluate models on Latin, CJK, and a handful of mid-resource scripts, but none treats script coverage as the primary axis of evaluation.

How the benchmark was built

GlotOCR Bench covers 158 Unicode scripts with two image variants per sentence: a clean rendering on a white background and a degraded rendering simulating aged documents. Sentences are sampled from real multilingual text and rendered into images, which are then presented to OCR systems for transcription. The total dataset contains 16,375 sentences.

The rendering pipeline is carefully engineered to be fair across scripts:

  • Text sourcing: The primary text collection is the GlotLID v3 dataset collection, which covers over 2,102 language-script pairs from different sources. Additional data was pulled from Wiktionary, WikiSource, Omniglot, and Common Crawl-derived datasets.