Skip to main content
Back

Peking University's DAP Framework Exposes How AI Cheats Math Olympiad Benchmarks

A new agentic framework strips the answer out of theorem-proving benchmarks and forces AI to find it first, revealing a huge reasoning gap.

  • DAP framework tackles Hard Mode theorem proving, where the answer must be discovered before being formally proven.
  • Sets SOTA: 36 solved on PutnamBench, 10 on CombiBench (previous best: 8).
  • Releases MiniF2F-Hard and FIMO-Hard, expert-reannotated benchmarks fixing ~35 semantic errors.
  • Reveals gap: LLMs guess correct answers 80%+ of the time, formal provers close under 10%.
  • Two-stage design (GPT-OSS-120B discovers, Goedel-Prover-V2 proves) prevents spurious rfl-based cheating proofs.
  • Self-verification loop with up to 30 iterations of self-criticism drives substantial gains on hard datasets.

Automated theorem provers have been quietly gaming their own benchmarks. When you ask a system to prove that the roots of an equation are exactly some set, and you already hand it that set inside the formal statement, you're testing something much easier than what a human competitor faces at an olympiad. A new paper from Peking University and Huawei calls this shortcut Easy Mode, and introduces both a benchmark and an agentic framework to fix it.

The framework, called Discover and Prove (DAP), sets state-of-the-art numbers on PutnamBench and CombiBench under the stricter setting. It also exposes a gap that benchmark design had been hiding: language models can guess the right answer to competition problems over 80% of the time, but formal provers close the proof under 10% of the time.

The answer was hiding in the question

The paper defines two settings for automated theorem proving (ATP) benchmarks:

  • Easy Mode: the final answer is baked into the theorem statement. The prover just has to verify a known claim.
  • Hard Mode: the answer is a sorry placeholder. The system must first discover what the answer is, then prove it.

The authors point out that embedding the answer in the statement reduces the intrinsic difficulty: for many solution-oriented problems, the primary challenge is discovering the answer rather than proving a consequence once known. The leaked answer often weakens the goal itself. In one IMO problem, the Easy Mode version only asked the prover to show that x falls in certain ranges, but not that every value in those ranges is actually attainable, quietly turning an if-and-only-if into a one-way implication.

Comparison of Easy Mode and Hard Mode formalizations of an IMO inequality problem in Lean 4

Cleaning up the benchmarks first