Epoch Caught 42% of FrontierMath Problems Broken, Sending GPT-5.5 Scores Soaring
Epoch AI overhauled FrontierMath after finding errors in 42% of problems, and the corrected scores reveal AI math ability is advancing faster than the benchmark itself
- Epoch AI released FrontierMath: Tiers 1–4 (v2), correcting errors found in 42% of problems after an AI-assisted audit.
- GPT-5.5 at xhigh reasoning effort now scores 85% on Tiers 1–3; Google's AI co-mathematician leads Tier 4 at 76%.
- Most errors were simple calculation mistakes by problem authors when extracting final answers; 12 problems were removed entirely.
- The audit was triggered by OpenAI, which funds the benchmark and holds exclusive access to ~80% of the dataset, flagging more errors than expected internally.
- Epoch AI says Tiers 1–4 is now approaching saturation and points to FrontierMath: Open Problems — genuinely unsolved research math — as the future of math benchmarking.
- The episode highlights a structural challenge: as models improve, hard benchmarks become harder to write and verify, and benchmark governance matters as much as benchmark difficulty.
FrontierMath: Tiers 1–4 (v2) is live, and the story behind it is as revealing as the scores themselves. Epoch AI has just published a sweeping correction to one of the field's most respected math benchmarks, after an AI-assisted audit uncovered errors in 42% of its problems. The rankings held up, but the scores jumped significantly across the board , and the episode raises hard questions about how we measure AI progress at the frontier.
A benchmark built for a different era
When FrontierMath launched in late 2024, current state-of-the-art AI models solved under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. The benchmark was designed to stay ahead of rapidly improving models: it is a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians, covering most major branches of modern mathematics , from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days.
Following the v2 update, the full FrontierMath dataset consists of 338 problems: a base set of 295 problems called Tiers 1–3, and an expansion set of 43 exceptionally difficult problems called Tier 4. Tier 4 is the hardest tier , Difficulty Tiers 1–3 cover undergraduate problems through exploratory problems suitable for an advanced graduate student, while Tier 4 is research-level mathematics.
The benchmark was developed through collaboration with over 60 mathematicians from universities across more than a dozen countries, spanning academic ranks from graduate students to faculty members, collectively holding 14 IMO gold medals, with one contributing mathematician also holding a Fields Medal.
How 42% of problems ended up broken
The audit was triggered by an uncomfortable disclosure. OpenAI , which funded the development of FrontierMath Tiers 1–4 and holds exclusive access to about 80% of the dataset , flagged internally that it had found more errors than expected during its own review. Epoch AI then launched an independent audit, using GPT-5.5 and Claude Opus 4.7 to flag potential errors, then engaging mathematicians to review those flags. Almost all were confirmed as real and severe.
The breakdown of what went wrong:
- Simple calculation mistakes accounted for the vast majority , off-by-one errors and flipped signs made when problem authors extracted the final answer
- Fatally ambiguous problem statements that made correct solutions impossible to determine
- More fundamental flaws in a smaller subset , 5 problems removed from Tiers 1–3 and 7 removed from Tier 4
v2 corrected 123 problems in Tiers 1–3 and 12 problems in Tier 4, and also removed 5 problems from Tiers 1–3 and 7 problems from Tier 4. The higher removal rate for Tier 4 reflects the greater complexity of those problems , more moving parts means more ways for a problem to be fundamentally broken.
The new leaderboard
With the corrected dataset, scores are higher across the board , not because models got better, but because some problems they were previously "failing" were actually unsolvable due to errors. The current standings:
- GPT-5.5 (xhigh reasoning effort): 85% on Tiers 1–3 , the top score on the base benchmark
- Google's AI co-mathematician: 76% on Tier 4 , the top score on the hardest tier
To understand how dramatic this shift is: OpenAI put FrontierMath near the center of its GPT-5.5 launch, reporting GPT-5.5 at 51.7% on FrontierMath Tiers 1–3 and 35.4% on Tier 4. Those were the pre-audit numbers. The jump to 85% on Tiers 1–3 is not a model improvement , it is what happens when you remove the problems that were impossible to solve correctly.
The "xhigh" label refers to GPT-5.5's maximum reasoning effort setting, which trades speed for depth of thinking. The xhigh reasoning effort introduces latency that is prohibitive for interactive use cases: Artificial Analysis measured a time-to-first-token of about 65 seconds at the highest setting, which is more than twenty times the median for non-reasoning frontier models.
Google's AI co-mathematician is a multi-agent system rather than a single model call. Google DeepMind introduced the AI co-mathematician as a multi-agent workbench designed to collaborate with human researchers on open-ended mathematical problems, built on the latest Gemini models. Unlike other systems evaluated with Epoch AI's standard agentic harness , which imposes hard token limits , the AI co-mathematician ran with no cap on model calls or tokens generated, meaning its inference cost is meaningfully higher. That caveat matters when comparing it to other entries on the leaderboard.
How models are actually evaluated
The evaluation setup is worth understanding if you want to interpret these numbers correctly.
For each FrontierMath question, the model needs to submit a Python function answer() that returns the answer , often an integer or a sympy object , and the implementation allows the model to reason and run Python code.
Scoring is all-or-nothing: no partial credit.
There is a hard limit of 1,000,000 tokens; if exceeded, the conversation ends immediately even if no answer has been submitted.
This means FrontierMath scores reflect math ability with access to computational tools, not pure pen-and-paper reasoning. A model that can write and execute Python to verify intermediate steps has a structural advantage over one that cannot.
The benchmark is approaching its ceiling
Epoch AI was blunt about what these scores mean for the benchmark's future: FrontierMath Tiers 1–4 is now approaching saturation. This is the same fate that befell every previous math benchmark. By 2024, frontier models were hitting 90%+ on the MATH benchmark. It had essentially been solved. When a benchmark gets saturated, it stops being informative.
The speed of this collapse is striking. A benchmark that launched with models solving under 2% of problems now has a model scoring 85% on its lower tiers , in roughly 18 months. The field is running out of hard-enough problems faster than it can write new ones.
What comes next: unsolved problems from real research
Epoch AI's answer to saturation is FrontierMath: Open Problems , a fundamentally different kind of benchmark. It is a collection of unsolved mathematics problems that have resisted serious attempts by professional mathematicians, where AI solutions would meaningfully advance the state of human mathematical knowledge.
The problems span number theory, combinatorics, algebraic geometry, and topology, rated by notability from "moderately interesting" to "breakthrough." Of the 15 currently listed problems, only one has been solved by AI so far , a Ramsey-style problem on hypergraphs, rated "moderately interesting." The hardest tier , problems rated "breakthrough,