KU Leuven Exposes How SemEval-2020 Corrupted an Entire AI Subfield's Leaderboard
A KU Leuven team audits the field's most-cited semantic change benchmark and finds OCR noise, mislabeled targets, and error bars so wide they swallow rankings.
ProTakeaways
- KU Leuven researchers audit SemEval-2020 Task 1, the dominant benchmark for lexical semantic change detection.
- Critique spans three axes: how change is defined, corpus data quality, and statistical robustness of the evaluation.
- Swedish data contains 212 OCR-noisy tokens per 100-row sample; English has mislabeled POS tags on gold targets.
- With only 31 to 48 targets per language, 95 percent error bars reach plus or minus 16 percentage points.
- Authors call for multi-dimensional, multi-timepoint benchmarks and transparent preprocessing logs.
- Correction code and analysis samples released on GitHub and Zenodo.
The benchmark that shaped almost every modern lexical semantic change detection system just got a hard look under the hood, and the results are unflattering. A team from KU Leuven has published a discussion paper auditing SemEval-2020 Task 1, the shared task that essentially defined how the field measures whether a word's meaning has shifted over time. Their conclusion: the benchmark isn't broken, but a lot of the leaderboard progress built on top of it may be less solid than it looks.
Lexical semantic change detection (LSCD) is the task of automatically identifying words whose meanings have shifted between two time periods. Think of how gay, cell, or broadcast moved through history. In NLP, the task is usually operationalised as a measurable difference in a word's usage across time-sliced corpora, where changes in the distribution of contexts count as evidence of semantic change. SemEval-2020 Task 1 became the dominant yardstick, attracting 33 teams and 186 systems across English, German, Latin, and Swedish.
Three cracks in the foundation
The paper attacks the benchmark from three angles: how it defines semantic change, how clean its data actually is, and how statistically meaningful its scores are.
On operationalisation, the benchmark models semantic change mainly as gain, loss, or redistribution of discrete senses, a framing too narrow to capture gradual, constructional, collocational, and discourse-level change. Sense boundaries in real language are fuzzy, and diachronic change tends to be gradient rather than a clean before/after snapshot. The "gold" labels themselves are downstream artifacts of annotator judgments, clustering algorithms, and threshold choices. Inter-annotator agreement was moderate at best (roughly 0.52 for English, 0.60 for German, 0.58 for Swedish), and the authors flag at least one target (the German noun