Universal NER v2 Exposes How GPT-5 Mini Hallucinates Entities Across 22 Languages
Universal NER v2 expands the multilingual entity-recognition benchmark to 22 languages and exposes major weaknesses in today's frontier LLMs.
PRO- Universal NER v2 expands the multilingual NER benchmark to 30 datasets across 22 languages and about 3 million tokens.
- Adds 11 new datasets in 10 languages including Greek, Hebrew, Japanese, Korean, Slovenian and Romanian legal text.
- Uses a three-tag schema (PER, LOC, ORG) inspired by Universal Dependencies for cross-lingual consistency.
- Frontier LLMs top out at 0.497 average F1 against 0.741 human inter-annotator agreement, exposing a big gap.
- GPT and Gemini over-annotate entities on parallel data by 30 to 90%, missing guideline nuance.
- Available on GitHub and Hugging Face under CC-BY-SA-4.0.
Named entity recognition looks solved if you only ever test in English. Push it into Korean legal text or Slovenian fiction and the picture gets messier fast, and until recently there was no clean, cross-lingually consistent way to measure how messy. The Universal NER project has been building that measuring stick since 2023, and its second major release widens the aperture considerably.
Universal NER v2 is a community-built, gold-standard benchmark that adds 11 new datasets in 10 typologically varied languages. The additions include several parallel evaluation sets aligned with each other and with existing UNER v1 corpora, all following the same annotation guidelines and inter-annotator agreement thresholds. Total footprint after v2: 30 datasets covering 22 languages and roughly 3 million annotated tokens.
Why entity recognition still needs a bigger yardstick
NER is one of the oldest tasks in NLP, yet multilingual evaluation remains fragmented. Different corpora use different tagsets, different annotation philosophies, and different domains, which makes it hard to tell whether a model is genuinely multilingual or just good at whatever happens to be in the test file.
UNER borrows a page from Universal Dependencies: one small, transferable tagset plus tight guidelines applied everywhere. The tags number just three, Person (PER), Location (LOC), and Organization (ORG), which sounds trivial until you have to decide whether "the White House" in a given sentence is a building or an institution. Annotation happens on the TALEN web tool, and to keep quality honest at least two annotators overlap on 5% of the data so inter-annotator agreement can be computed.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
