Six Frontier LLMs Independently Caught the Same Fake 99% Accuracy
Six frontier language models independently audited a published gesture-recognition paper and all converged on the same verdict: the 99% accuracy came from data leakage.
ProTakeaways
- A new arXiv study tests whether LLMs can independently detect data leakage in published ML papers.
- Six frontier models (GPT-5.2, Claude Sonnet 4.6, Gemini 3.0 Pro, Kimi 2.5, DeepSeek-V3, GLM-5) audited the same paper blind.
- All six converged on the same diagnosis: subject-level leakage from a random 90/10 split over only six participants.
- Models cited the same three fingerprints: synchronized train/test curves, minimal generalization gap, near-perfect confusion matrix.
- The target paper reported 99% gesture recognition accuracy but likely measured within-subject memorization, not generalization.
- Suggests LLMs could serve as cheap first-pass auditors for common evaluation flaws in peer review.
A new arXiv preprint takes a provocative angle on the reproducibility crisis: rather than asking humans to spot methodological flaws in published machine learning work, why not ask the models themselves? The study by Domonkos Varga handed a single suspicious paper to six frontier LLMs, gave each an identical prompt, and checked whether they could independently diagnose data leakage without any hint about what was wrong.
They all did. Every model flagged the same flaw, cited the same evidence, and reached the same verdict.
The paper on the operating table
The case study is a 2021 paper titled Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning. The system uses OpenPose to extract 2D coordinates of 18 body joints per detected person from aerial video, then feeds those skeletal features into a downstream classifier. Ten gestures were defined, including Kick, Punch, Squat, Stand, Walk, and Sit, plus two application-specific dynamic gestures, Attention and Cancel, used to initiate and terminate UAV interaction.
The dataset was collected from six participants, each performing multiple variations of every gesture. Skeletal data were extracted frame-by-frame to form the training samples. A four-layer fully connected network (128-64-16-10) was trained for 50 epochs with a 90/10 split, and the authors reported roughly 99% accuracy on both training and test sets, with an almost perfectly diagonal confusion matrix.
