Skip to main content
Back

49 Mathematicians Built a Benchmark That GPT-5.5 Already Nearly Solved

49 mathematicians crafted 100 research-level math problems; only 2 stumped the best LLMs after extended thinking runs

Paper, Full Access
  • 49 research mathematicians built a 100-question benchmark of research-level math problems at a workshop hosted by the Max Planck Institute in Leipzig.
  • Evaluation ran in 3 stages; only 2 of 100 questions remain unsolved after extended-thinking model runs.
  • GPT-5.5 solved 44 questions in a single shot (Stage 1); with extended thinking across 3 runs, it solved 88 of 100.
  • The authors conclude that exercise-style benchmarks grounded in published research have hit their ceiling for the strongest models — future hard benchmarks may need unpublished or open problems.
  • AI-assisted review corrected 3 answers originally submitted by human mathematicians, highlighting LLMs' emerging role in peer review of mathematical content.
  • All 100 questions are publicly available via the ScienceBench platform and in the paper's appendix.

A group of 49 research mathematicians has published the Leipzig Benchmark, a curated collection of 100 genuinely hard mathematics questions designed to probe the outer limits of what large language models can actually reason about. The result is both a new benchmark and a candid progress report: the best models today are impressively capable, but the frontier is closing faster than most researchers expected.

Where the problems came from

Between April 1 and May 15, 2026, the group compiled a dataset of research-level mathematics questions with known answers, with most of the work done during a 3-day workshop at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany. The workshop brought together 35 on-site participants and 14 remote contributors, spanning fields from Algebraic Geometry and Matroid Theory to Knot Theory and Probability.

The questions were not textbook exercises or competition problems. Contributors were asked to create examples rooted in existing, publicly accessible research but going beyond what could fit in a paper. Think: a lengthy worked example, a specific numerical invariant that requires deep domain knowledge to compute, or a combinatorial count that demands understanding a recent theorem. Each question had to have a unique, well-defined, unguessable answer.

The benchmark spans 25 research areas, with the heaviest concentration in:

  • Algebraic Geometry (36 questions)
  • Algebraic Combinatorics (22 questions)
  • Matroid Theory (18 questions)
  • Representation Theory (15 questions)
  • Enumerative Combinatorics (15 questions)

The three-stage evaluation

The evaluation ran in three stages: a single attempt by five state-of-the-art LLMs, followed by a 20-runs-per-model evaluation with three of these models, and finally a 3-run attempt with two heavy-thinking models.