Artificial Analysis Rebuilds Its AI Leaderboard to Stop Models From Gaming It

The leaderboard many teams use to pick their default frontier model just got harder to game. Artificial Analysis has rolled out Intelligence Index v4.2, an interim refresh pulling forward pieces of the planned v5 to keep pace with a suddenly crowded frontier. The headline changes: more private test sets, longer and messier real-world tasks, and the retirement of a benchmark that models have effectively solved.

The Index is a weighted composite score across nine evaluations, split into Agents (34%), Coding (24%), Scientific Reasoning (24%), and General (18%). v4.2 keeps that skeleton but swaps in two new evaluations and drops one that no longer discriminates between models.

What Changed in v4.2

• GPQA Diamond is out. The graduate-level science multiple-choice test has been saturated by frontier reasoning models and no longer separates them.
• AA-Briefcase is in. A private, in-house agentic knowledge-work eval built by industry experts, testing multi-week projects with thousands of source files per scenario.
• GDP.pdf is in. A long-context document reasoning benchmark from Surge AI covering 4,592 PDF pages across 100 tasks and ten professional domains.
• 40% of the Index is now held-out, double the private share in v4.1, drawing on AA-Briefcase, AA-Omniscience, and CritPt solutions.
• Grading infrastructure hardened across AA-LCR, GDPval-AA, and SciCode to fix answer-key errors, re-anchor Elo scales, and stop counting slow-but-correct code as a failure.

GDP.pdf: Punishing by Design

GDP.pdf tests whether a model can answer realistic domain questions about specific PDFs rather than measuring OCR, table QA, or chart reasoning in isolation. Built around documents that run professional work — benefits packets, leases, datasheets, clinical guidelines, construction plans — it covers 100 tasks across ten domains, with 1,275 rubric criteria in total, averaging about thirteen graded requirements per task. Artificial Analysis reports an All-pass Rate that credits a task only when every criterion is satisfied.

In the original Surge paper, the best model passed only 15% of items and the worst passed 1%, with errors tracing back to misaligned tables, misread charts, skipped footnotes, miscounted floor-plan symbols, scan noise, and amendments that supersede earlier text. Newer frontier models do better but remain well below human-level: on the v4.2 run, GPT-6 Astra leads at 33.2%, GPT-5.6 Sol follows at 28.2%, and Claude Fable 5.1 sits at 26.2%.

AA-Briefcase and the Anti-Gaming Push

AA-Briefcase addresses the fact that public benchmarks eventually leak into training data. Each scenario is a multi-week business project with two to five tasks per week. The agent runs in a sandboxed Linux environment with no internet access, up to 500 turns, and a code-execution tool. Grading combines binary rubric checks with pairwise Elo comparisons on analytical quality and presentation, judged by a rotating panel of Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro Preview to reduce same-family bias.

Source pools are designed to resemble actual work: Slack exports, spreadsheets, PDFs, interview transcripts, market research, standards documents, app-store pages, board materials, and emails. Because the test set stays private, labs cannot optimize directly against it.

What the New Leaderboard Shows

Updated rankings put Claude Fable 5.1 at the top, followed by GPT-6 Astra, which shows a 4-point gain over GPT-5.6 Sol. Meta, SpaceXAI, Moonshot/Kimi, Z AI, and Google round out the leading labs. On AA-Briefcase specifically, Claude Fable 5.1 and Opus 5 lead, with GPT-6 Astra roughly 85 Elo points above GPT-5.6 Sol.

The cost-per-task Pareto frontier is now shared between Anthropic, OpenAI, Meta, and Z AI. Token efficiency tells a different story: GPT-6 Astra dominates output-token efficiency near the intelligence frontier, while Claude Fable 5.1 and Gemini 3.8 Flash burn the most tokens among models scoring above 25 on the Index.

Held-Out Data as the Real Signal

Surge has noted that no frontier model exceeds 25% on its harder agent benchmarks, with agents firing employees without authorization, approving self-submitted expenses, and sending expired medical records to insurers. Combined with GPQA Diamond saturating, the pattern is clear: multiple-choice academic tests have lost their usefulness as a signal for frontier progress, and long-horizon agentic tasks on messy real documents now do the separating. Artificial Analysis has signalled the private share will grow again in v5.

For teams using the Index to pick production models, weighting the agentic and long-context columns more heavily than the composite score reflects where discrimination between models actually lives — and the leaderboard will keep reshuffling as private evals expand.