ChatGPT Pro Caught a Nobel Laureate's 40-Year Math Error With Help
A new paper tests frontier AI models on published economic theory proofs with known errors — and finds they all fail without human guidance
ProTakeaways
- A new paper by Alexis Akira Toda tests Gemini, Claude, Refine, and ChatGPT on four published economics papers with known proof errors.
- No model found a true error without substantial human guidance — every model initially said the proofs were correct.
- ChatGPT 5.5 Pro ($100/mo) was the clear winner, constructing valid counterexamples and corrected proofs once pointed in the right direction.
- The primary test case was a 40-year-old error in Tirole (1985), a Nobel laureate's paper cited 2,000+ times.
- Data contamination is a key confound — the Tirole test was run with memory/web search disabled to ensure genuine reasoning.
- The paper argues a human + frontier AI already outperforms typical peer review, where referees rarely read proofs.
A few months ago, a paper published in Econometrica quietly dropped a bombshell: Proposition 1(c) of Jean Tirole's 1985 classic "Asset Bubbles and Overlapping Generations" , a paper cited over 2,000 times and written by a Nobel laureate , was mathematically incorrect. Word spread fast in economics circles, and many assumed AI must have helped find the flaw. It didn't. Now, the economist who co-authored the correction has run a systematic experiment to find out whether AI could have found it , and the answer is a qualified no.
The Setup: A Clean Experiment With a Known Answer
Alexis Akira Toda of Emory University published a paper documenting his experiments feeding four published economic theory papers , each containing a known error he helped identify , to several frontier AI models: Gemini 3.5 Flash, Refine, Claude Opus 4.8, and ChatGPT 5.5 Pro. The key advantage of this setup is that Toda already knew exactly where each proof broke down, putting him in the best possible position to prompt and steer the models.
The primary test case was Tirole (1985). Tirole's Proposition 1(c) claims that if the dividend growth rate is above the bubbleless interest rate but below the population growth rate, then there exists no bubbleless equilibrium , but the correction shows this result is incorrect by presenting a counterexample economy that satisfies all of Proposition 1(c)'s assumptions yet has a unique bubbleless equilibrium. The error had survived peer review for over 40 years.
To make the test as clean as possible, Toda ran the Tirole experiment with ChatGPT's memory and web search disabled, since the correction had only just been published. This guarded against the most dangerous confounder in any AI evaluation: data contamination , the possibility that a model simply retrieves a known answer from its training data rather than reasoning to it from scratch.