Artificial Analysis Moves Intelligence Index to 60% Private Tests to Stop AI Cheating

Artificial Analysis has released version 5 of its frontier-model Intelligence Index, adding two benchmarks, upgrading three, and assigning 60% of the score to private evaluations for the first time. The revised Intelligence Index methodology aims to reduce training-data contamination, which can inflate results when benchmark questions or answers circulate online.

A companion Progress Index methodology addresses historical comparisons. It places models released since November 2022 on a shared scale, allowing researchers to compare generations even as individual benchmarks change.

Private tests take 60%

Private evaluations now provide most of the Intelligence Index score. Their questions, answer keys, or both remain off the public internet, reducing the chance that benchmark material appears in model training data.

Agents, Coding, Scientific Reasoning, and General contribute 30%, 20%, 20%, and 30% of the composite score, respectively. Agentic evaluations, which require models to use tools and complete multistep tasks, account for 55% of the score and a median 88% of generated output tokens. Terminal-Bench-Science alone accounts for roughly one-third of output tokens generated per Index task.

Claude leads, specialists split the wins

Anthropic and OpenAI occupy all five leading positions, with Gemini 4 Argon next at 50. Leadership varies across the nine evaluations: Claude Opus 5.5 leads three, while five other models each lead one. Model selection therefore depends on the workload represented by each evaluation.

Knowledge questions produced lower scores than reasoning questions throughout HLE-Diamond. Claude Opus 5.5 and GPT-6 Astra each scored 78% on reasoning, then fell to 58% and 50%, respectively, on knowledge.

Released weights trail by 26 points

The Index’s reported gap between the leading closed model and its highest-ranked open-weights model grew from 3 points in v4.3 to 12 points in v5. That comparison includes Kimi K3.1 Pro, whose weights have not yet been released. Among models with downloadable weights, the gap reaches 26 points.

The largest gaps appear on the hardest new evaluations. Kimi K3.1 Pro scores 37% on HLE-Diamond, compared with 68% for Claude Opus 5.5. It reaches 34% on Terminal-Bench-Science, compared with GPT-6 Astra’s 63%. The best model with released weights scores 13% on that science benchmark, and most open models remain below 10%.

Near-top scores span a 13-fold cost range

Among the ten highest-scoring models, reported cost per Intelligence Index task ranges from $0.96 for GPT-6.1 Terra to $13.04 for Claude Fable 5.1. GPT-6.1 Sol scores 55, three points behind Claude Opus 5.5, at about one-eighth of the per-task cost: $1.27 compared with $10.04.

Effort settings now produce wider quality and cost ranges. These settings control how much inference-time computation a model uses for a task.

• Claude Opus 5.5: Its score rises from 35 at low effort to 58 at maximum effort, with a tenfold increase in cost.
• Claude Sonnet 5.5: Its low-to-maximum spread reaches 29 points, up from 20 in v4.3.
• GPT-6.1 Sol: Its spread doubles from 10 points to 20.

Every tested model generates more output than it did under v4.3. Median output per Index task rose by about 69%, from 35,000 to 59,000 tokens. Those additional tokens can increase both latency and inference spending, depending on the provider’s pricing.

Changing tests get a stable scale

The Intelligence Index periodically adds evaluations that challenge current models and retires those that stop separating leading systems. Those revisions improve current comparisons but place versions such as v3 and v5 on different raw scales.

The new Progress Index results use item response theory, a statistical method drawn from psychometrics. Shared evaluation tasks act as bridges between models, allowing the system to estimate performance on one scale even when models completed different test sets.

The model estimates contain three main elements:

• Capability: one overall estimate for each model.
• Difficulty: the capability level at which a model has a 50% expected score after adjustment for guessing.
• Discrimination: the rate at which expected performance rises as capability exceeds an item’s difficulty.

Artificial Analysis anchors the scale with o3-mini at high effort set to 220 and GPT-5.5 at extra-high effort set to 320. These values provide stable reference points and have no percentage unit.

The methodology cites prior work from Epoch AI’s Capabilities Index and Anthropic’s capability-scale research. Coverage extends to language models released from November 2022 onward, supporting quantitative comparisons of capability gains between successive model generations.

Two indexes serve different decisions

• Use v5 as a new baseline because its raw scores are not directly comparable with earlier Intelligence Index versions.
• Inspect evaluation-level results for workload fit; the overall leader does not lead every category.
• Distinguish announced open-weights models from models whose weights are available for deployment.
• Include effort settings, output-token volume, latency, and per-task cost in production estimates.
• Validate shortlisted models on internal tasks. Private tests reduce exposure to leaked material, but restricted access also limits independent replication.

A defensible selection process uses the Intelligence Index to identify candidates, evaluation-level results to match them with workloads, and internal tests to measure production quality, latency, and cost. The Progress Index supplies the corresponding view of capability growth over time.