Epoch AI flags nine of 15 AI benchmarks as flawed

Epoch AI has launched Benchmark Reviews, a third-party audit program that examines whether AI evaluations support the claims attached to their scores. Its first audit results classify four benchmarks as Verified, nine as Flawed, and two as lacking enough public information for review.

Benchmarks influence model launches, procurement, research, and safety claims. Errors in tasks, scoring code, or evaluation settings can inflate or suppress scores, alter rankings, and make results from different model runs difficult to compare.

A pass requires every check

Epoch assigns one of three labels using a published rubric. Each verdict describes whether the benchmark can be interpreted as its creators claim.

Verified is narrower than flawless. The label means Epoch found no issue substantial enough to invalidate the benchmark’s intended interpretation under its rubric.

How a benchmark fails the audit

Epoch first applies a reviewability check covering tasks, scoring logic, and harness settings. A harness is the software environment that runs and scores a model, including its token limits, reasoning effort, available tools, turn limits, and other configuration choices. Missing access can prevent a meaningful audit.

Reviewable benchmarks then enter a scoring assessment where failure on any criterion produces a Flawed verdict. The default quantitative threshold is an error rate of at least 20% in the inspected sample, although a systemic grading problem can also trigger the label.

Benchmarks containing more than 50 tasks receive a random sample of 50, stratified by category. If the observed error rate falls between 15% and 25%, Epoch expands the sample to 100 tasks. Smaller benchmarks are reviewed in full.

Reviewers examine several recurring sources of misleading scores:

• Invalid tasks: Problems that are impossible as written because requirements, files, or other necessary information are missing.
• False negatives: Correct responses rejected by rigid scorers, stale reference answers, or sandbox failures unrelated to the model.
• False positives: Incorrect responses accepted by permissive scorers, exploitable environments, or answers exposed through the harness or web.
• Under-elicitation: Token, time, or turn limits that prevent a model from demonstrating the capability under evaluation.
• Setup bias: Unequal compute budgets or scaffolding tuned for only a subset of models.
• Silent versioning: Changes to scorers or reference answers without a version update, causing leaderboards to combine incomparable runs.

Headline scores need their audit trail

Several affected benchmarks measure capabilities central to developer tooling. SWE-bench Verified evaluates coding agents on real GitHub issues, Terminal-Bench tests work performed through a terminal, and Berkeley Function Calling Leaderboard evaluates tool and function use. Problems in their tasks or graders can distort comparisons among coding models and agent frameworks.

A Flawed label identifies a documented interpretation problem. Some results may remain useful when the affected tasks, scoring errors, and evaluation settings are understood. Epoch publishes a concise explanation for each Flawed finding and a fuller assessment for Verified benchmarks, including a limitations section.

Reliable model comparisons therefore require more than a benchmark name and aggregate score. The benchmark version, harness configuration, model settings, task sample, and known scoring failures all affect what the number measures.

The audit has defined boundaries

Epoch excludes its own benchmarks from the program because reviewing them would create a conflict of interest. Its documentation cites a separate analysis that found errors in 42% of FrontierMath v1 problems, showing why benchmark ownership and quality review need separate treatment.

Private benchmarks can receive confidential reviews. Epoch says creators may provide tasks without making them public, while the resulting report discloses generic categories of scoring errors rather than task-level details. This arrangement preserves test secrecy while limiting how much outside researchers can independently verify.

Each verdict applies to a specific benchmark version. Creators can correct reported problems and request another review, while score comparisons remain tied to the tasks, grader, and configuration that Epoch inspected.

Coverage will follow impact and reach

Epoch plans to select future audits using three criteria:

• Impact: Benchmarks covering consequential tasks or safety-relevant capabilities.
• Reach: Widely cited evaluations and benchmarks used in recent model or system cards.
• Diversity: Capabilities that remain underrepresented in the reviewed collection.

Developers and researchers can suggest additional benchmarks to Epoch’s review team. With 11 of the first 15 evaluations either Flawed or insufficiently documented, benchmark review status now provides material context for model selection, procurement, and capability research.