Skip to main content
Back

Renmin University's OlymMATH Exposes Gemini 2.5 Pro's Shaky Math Reasoning

A new bilingual Olympiad math suite exposes how the strongest reasoning models still stumble, guess symmetries, and lose accuracy in Chinese.

Pro

  • OlymMATH releases 350 Olympiad problems in parallel English and Chinese, sourced from printed publications to minimize contamination.
  • Three subsets: EASY, HARD, and LEAN, spanning numerical answers and Lean 4 formal proofs.
  • Gemini 2.5 Pro reaches 58.4% on HARD-EN, o3-mini 31.2%, DeepSeek-R1 just 19.5%.
  • All models score higher on English than Chinese, statistically significant across subjects and difficulty.
  • Specialist Lean provers score around 10%, versus 80% on miniF2F, exposing a formal reasoning gap.
  • Team released 582k reasoning trajectories from 28 models, a visualization tool, and expert solutions.

Math benchmarks keep getting solved. AIME 2024 is already near-ceiling for frontier models, and OlympiadBench falls to a 1.5B parameter model. A team from Renmin University set out to build something that resists the current wave of reasoning models, and their new suite makes even Gemini 2.5 Pro look shaky.

OlymMATH is a 350-problem, English-and-Chinese Olympiad math suite built to do two things at once: score final numerical answers with a rule-based checker, and verify full proofs through the Lean 4 theorem prover. The dataset, evaluation code, 582,400 reasoning trajectories from 28 models, and a visualization tool are all open source.

Why existing math benchmarks stopped being useful

Two evaluation paradigms exist for math reasoning, and each has broken in its own way. Numerical-answer benchmarks using rule-based verification (e.g., sympy) offer scalability but cannot assess reasoning quality, while formal proof benchmarks using theorem provers (e.g., Lean, Isabelle) provide rigorous process-level verification at the cost of heavy formalization work.

The scale problem is real. A single problem shift on AIME can change accuracy by 3.33%, with binomial standard errors roughly 2.6 times larger than those from a 200-problem benchmark. Larger sets like OlympiadBench have the opposite issue: even a 1.5B model (DeepScaleR-Preview) reaches 50.0% accuracy, indicating insufficient challenge for evaluating state-of-the-art reasoning systems. Contamination is another headache. Suites that scrape AoPS or reuse IMO problems risk showing up in pretraining corpora, inflating scores.

How the benchmark is built

OlymMATH takes a manual, offline route. The 350 problems were curated from printed resources (specialized magazines and textbooks), intentionally excluding online repositories to minimize prior digital exposure. A China Mathematical Olympiad silver medalist and two provincial first-prize winners verified the problems and solutions.