StudentBench Shows AI Tutors Match Human GRE Gains at 918x Lower Cost
A new benchmark study finds top AI tutors match expert human GRE tutors on learning gains, at up to 918 times lower cost.
- StudentBench benchmarks 13 AI tutors against expert human tutors on real GRE learning gains with 2,383 participants.
- AI tutoring was statistically equivalent to human tutoring (p = .015) across the full study.
- Best AI tutor beat human tutors on 5 of 7 GRE domains on average.
- Cheapest effective AI tutor cost 918x less per percentage point gained ($0.0052 vs $4.81).
- Faster AI reply times drove more engagement, more practice, and larger learning gains.
- Platform, dataset, and code are publicly available.
StudentBench Finds AI Tutors Can Match Human GRE Gains
Researchers behind StudentBench report that leading AI tutors produced GRE learning gains statistically equivalent to expert human tutors in a study of 2,383 participants. The result has defined limits: it covers a one-hour GRE preparation session, and performance varied sharply across tutors and subject areas.
StudentBench combines a public evaluation platform with a study of 13 AI tutors built from frontier models supplied by multiple vendors. Curtis Northcutt led the research paper with Inaara Hasmani, Kevin Feng, Trevor Khangi, Andreas Plesner, and Jonas Mueller. Handshake AI Research, the jobs platform’s data arm, provided the funding.
Why Answer Accuracy Misses the Lesson
Conventional LLM leaderboards measure whether a model can solve GRE questions. StudentBench measures whether a student improves after interacting with that model, using pre-test and post-test scores to quantify learning.
Static question-answer accuracy cannot establish teaching quality. Previous evaluations have found substantial differences among AI tutoring products, while a separate Stanford study found limited student use when AI tutoring was merely made available. Effective tutoring also depends on lesson structure, practice quality, responsiveness, and sustained participation.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
