Weizmann Researchers Built a Tool That Spots Which Exam Questions AI Aces
Researchers borrow a psychometric bias detection technique to pinpoint exactly which exam questions favor chatbots over students, and vice versa.
Takeaways
- Weizmann researchers adapt Differential Item Functioning to detect questions where chatbots and humans systematically diverge.
- Method tested on six leading LLMs including GPT-5.2, Gemini 3 Pro, and Claude 4.5 Sonnet.
- Logistic Regression DIF beats Mantel-Haenszel, producing zero false positives on placebo runs for the psychometric exam.
- Item-total correlation of 0.20 recommended as a quality filter to remove noisy items upfront.
- Chatbots excel at rule-based conceptual reasoning; students win on visual interpretation and multi-step calculations.
- Code and sample data released for reproducing the analysis on other assessments.
Educators have a growing problem. Students are quietly routing homework and exams through large language models, and the standard defense of comparing average scores says almost nothing about which specific questions are actually vulnerable. A team from the Weizmann Institute of Science just published a method that fingerprints individual test items where chatbots and humans systematically diverge, borrowing a statistical tool originally built to catch demographic bias in standardized tests.
The approach is called Differential Item Functioning (DIF) analysis, traditionally used to detect bias across demographic groups. The twist: treat chatbots as one demographic and human learners as another, then ask a precise question for every item on a test. Given two respondents with the same overall ability, does one group have a systematically different probability of getting this specific question right?
Why raw score comparisons fall short
Most current LLM assessments report descriptive benchmarks: average accuracy, question-type breakdowns, headline pass rates. That misses the interaction between task difficulty and respondent ability. A question could have identical average success rates for humans and bots while hiding the fact that low-ability students beat low-ability chatbots on it, even as high-ability chatbots crush high-ability students. DIF surfaces exactly that kind of pattern by conditioning on ability rather than aggregating over it.