NLP's Dirty Secret: Annotated-llm Exposes 7 Years of Missing Human Data
A new audit of 1,603 NLP papers finds that who annotates, how they're trained, and what they're paid is routinely left out of published research
- First large-scale audit: Researchers audited 1,603 ACL-venue papers (2018-2025), extracting 2,667 annotation tasks to assess what gets reported about human annotators.
- Critical details routinely missing: Compensation, annotator training, language proficiency, socio-demographics, and inter-annotator agreement are frequently omitted from published papers.
- Model evaluation worst offender: Papers using humans to judge model outputs report far less about their annotation setup than papers building datasets.
- LLM pipeline validated: An LLM-assisted extraction system matched human-level agreement (Krippendorff's alpha 0.606 vs. 0.585 human-human) on a gold-standard benchmark of 41 papers.
- ACL checklist helped, but unevenly: The 2022 ACL Responsible NLP Checklist improved reporting over time, but gains are inconsistent across venues and topics.
- Taxonomy + datasets released: The paper introduces a 25-category reporting taxonomy and releases both Annotated-gold and Annotated-llm datasets with bare-minimum reporting recommendations.
Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled. That gap is not just a paperwork problem. When you can't reproduce an annotation setup, you can't trust the benchmark, and when you can't trust the benchmark, you can't trust the model scores built on top of it. A new paper from researchers at the University of Technology Nuremberg and the Interdisciplinary Transformation University Austria takes the first systematic look at just how bad this problem really is.
Seven years of silence on the basics
The team provides the first large-scale, task-level audit of human annotation reporting across major NLP venues, asking which annotation details are documented, which are missing, and how reporting varies across time, topic, venue, and intended use of human judgment. The scope is significant: they construct Annotated-llm, a dataset covering ACL-venue papers from 2018 to 2025, with 2,667 extracted annotation tasks from 1,603 papers.
The results are a mixed bag. Papers are reasonably good at reporting the operational basics -- things like how many annotators were used, where they were recruited, and what kind of expertise they had. But the details that actually let you assess whether the labels are valid? Those are routinely missing.
Worryingly, there is extremely little empirical evidence about the annotators in the NLP community: whether the community uses crowdworkers or experts, whether annotators are adequate for the task, and how well they are compensated. There is also a lack of quantitative evidence on whether initiatives such as the ACL Responsible NLP Checklist, introduced to improve consistency and transparency, correspond to measurable improvements in annotation reporting quality.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
