Epoch Hides a Game's Identity to Stop AI Labs From Cheating Benchmarks
Epoch AI's new Mystery Game Puzzles benchmark hides its game identity to stop models from gaming the test -- Opus 5 leads at 59%
- New benchmark launched: Epoch AI's Mystery Game Puzzles tests AI on 100 positions from an undisclosed game to prevent targeted post-training.
- Current leader: Opus 5 holds the record at 59%, with GPT-5.5 previously reaching 56% -- scores have plateaued since April.
- Open-weight milestone: Qwen3.8-Max scores 38%, the best open-weight result, edging out GPT-5.4 and Opus 4.8.
- Compute isn't the bottleneck: Models get a 1M output token budget but top models use under 400K -- failures are reasoning failures, not resource limits.
- Anti-contamination by design: The game's identity, prompts, and transcripts are all withheld to prevent benchmark-specific tuning.
- Progress stalled: Scores jumped from 25% to 56% in two months early in the year, but have been roughly flat since then.
Benchmarks have a contamination problem. The moment a benchmark is public, labs can -- intentionally or not -- tune their models toward it. Epoch AI's answer to that is a new benchmark called Mystery Game Puzzles: 100 positions from a well-known game whose identity is deliberately kept secret, so models can't be specifically prepared for it.
The benchmark with no name
The benchmark consists of puzzles based on a well-known game where models must select the single best next move. It is similar in setup to Epoch's Chess Puzzles benchmark, except that the game's identity is kept secret to reduce the risk of benchmark-specific preparation. That secrecy extends to the prompts, example positions, and model transcripts -- none of which are currently published.
The design logic mirrors what Epoch already does with Chess Puzzles. While the broader significance of chess is low, Epoch finds the format useful because it plausibly measures important aspects of reasoning, including spatial reasoning and planning -- a "lite" version of more involved video game benchmarks. The mystery variant takes that one step further: by hiding the game, it removes the possibility that a model was specifically post-trained (fine-tuned after pretraining, often on curated reasoning examples) on that exact task.