Epoch Hides a Game's Identity to Stop AI Labs From Cheating Benchmarks

Benchmarks have a contamination problem. The moment one goes public, labs can tune their models toward it, intentionally or not. Epoch AI's answer is Mystery Game Puzzles: 100 positions from a well-known game whose identity is deliberately withheld, so models can't be specifically prepared for it.

A benchmark with no name

Models must select the single best next move from each position. The setup mirrors Epoch's existing Chess Puzzles benchmark, but the game's identity is kept secret to block benchmark-specific preparation. That secrecy extends to the prompts, example positions, and model transcripts, none of which are currently published.

The reasoning behind the format is the same as with Chess Puzzles: the game plausibly tests spatial reasoning and planning without requiring the elaborate infrastructure of full video game benchmarks. Hiding the game adds one more layer. If a model was never post-trained on this specific task, its score reflects general reasoning ability rather than task familiarity. Post-training refers to fine-tuning a model after pretraining, often on curated reasoning examples, which can inflate scores on known benchmarks without improving underlying capability.

What the scores show

Progress was fast early in the year, then stalled:

• Opus 4.6 scored 25%, the starting baseline
• GPT-5.5 jumped to 56% two months later
• Opus 5 currently holds the record at 59%
• Top scores have been roughly flat since April

The open-weight result deserves attention separately. The highest score from an open-weight model is 38%, from Qwen3-8B-Max, slightly ahead of GPT-5.4 and Opus 4.8. Epoch reads this as a meaningful signal: open-weight models are gaining ground on out-of-distribution tasks, not just catching up on familiar benchmarks.

Tokens aren't the bottleneck

Each model receives a 1 million output token budget per puzzle, but top performers don't come close to using it. Opus 5, for example, uses more tokens on puzzles it gets wrong, but never more than 400K. Models that fail aren't running out of thinking room. The reasoning itself breaks down.

There's also a scaffolding detail worth knowing. Some models hit their single API-call output token limit mid-puzzle. When that happens, Epoch feeds the cut-off reasoning back to the model and asks it to continue, compacting as necessary. Most models finish within three such turns.

Why secrecy is the design

Contamination happens when models encounter benchmark questions, or close variants, during training. Scores rise without any real improvement in reasoning. Mystery Game Puzzles tries to close that gap. A model that has never seen training data specific to this game's puzzle format produces a score that reflects raw reasoning rather than memorized patterns.

Epoch applies a similar principle to Chess Puzzles, where all positions are generated programmatically and don't appear in any other source. Each has a single best move as judged by the Stockfish engine, and models are graded on whether they find it. The mystery variant runs the same logic but withholds even the game's name.

What the plateau means

Going from 25% to 56% in two months on a task with no specific post-training is a real jump. The ceiling at roughly 59% since April suggests that general-purpose reasoning, however improved, doesn't transfer cleanly to novel structured domains. The headroom is real and currently untouched.

The benchmark is free to follow on Epoch's benchmarking hub, which covers dozens of tasks across math, coding, agents, and games. The leaderboard updates as new models are evaluated, and all data is available under a Creative Commons Attribution license.