LMU Munich Finds GPT-5.2 Labels Beat Human Annotators at One-Tenth the Cost
A study on German TikTok hostility detection finds GPT-labeled data trains stronger classifiers than crowdworker labels at one-tenth the cost.
Takeaways
- Classifier trained on 25,974 GPT-5.2 labels ($43) beats one trained on 3,800 human labels ($316) at ~10x lower cost.
- Result holds for closed-source GPT-5.2 and open-weight Qwen3.5-122B, and under Bayesian soft-label evaluation.
- The gain comes from a two-question interface (topic reference + valence) that matches the human annotation task.
- Active learning provides no reliable advantage over random sampling in a pre-filtered pool, across four acquisition functions.
- LLM-supervised classifiers produce errors at much higher confidence: 52-78% above 0.9 vs 10-38% for human-trained.
- Border Control and Islam clusters are over-flagged by every LLM regardless of prompt or model choice.
A new study from LMU Munich and the University of Copenhagen puts a hard number on a question many teams have been circling: if a modern LLM can annotate an entire corpus for pocket change, is there still any reason to run active learning with human labels? Working on anti-immigrant hostility detection in German political TikTok comments, the authors find that a classifier trained on GPT-5.2 labels beats one trained on Prolific crowdworker labels at roughly one-tenth the cost, with a catch that matters for anyone shipping a moderation pipeline.
A labeling bake-off on 278K comments
The authors built a fresh corpus by pulling public comments from the official TikTok accounts of six major German political parties (CDU/CSU, Bündnis 90/Die Grünen, SPD, FDP, AfD, Die Linke) via the TikTok Research API, covering January 2024 to September 2025, yielding 467,660 comments. After deduplication and language filtering with GlotLID, 277,902 German comments remained.
A two-stage LLM pipeline narrowed things down. Llama-3.3-70B acted as a cheap high-recall filter for immigration-relevant content, reducing the pool to 25,974 comments. Those were then labeled by two annotator LLMs at Stage 2: closed-source GPT-5.2 and open-weight Qwen3.5-122B-A10B, a mixture-of-experts model with roughly 10B active parameters.
In parallel, six Prolific crowdworkers annotated 5,000 of those comments, producing 1,200 multiply-annotated items as a gold evaluation set and 3,800 single-annotated items as a training pool. All classifiers, whether human- or LLM-supervised, were evaluated head-to-head on the same held-out human gold set across four German encoders (ModernGBERT, german-bert, gbert-base, and multilingual xlm-r-base) with 10 random seeds each.
The two-question trick that changes everything
The most important design choice in the paper has little to do with LLMs. It concerns the annotation interface. An early holistic pilot that asked annotators "is this comment anti-immigrant, yes or no?" collapsed with a Fleiss kappa of 0.175, indicating very low agreement. Humans could not consistently draw the line between hostility toward immigrants and criticism of immigration policy.