Artificial Analysis Rebuilds Decision Index v2 to Reward AI Confidence
Artificial Analysis rebuilt its Decision Index around a calibration-aware metric, private document benchmarks, and symmetry tests that penalize models for wobbly confidence.
- Decision Index v2 launched with calibration-aware headline metric that rewards both correctness and confidence.
- Private datasets now make up 45% of the index, up from 25%, to fight benchmark gaming.
- Drex Max v1.1 and GLiDE v1.2 lead decision models at 73; Claude Haiku 5.5 tops overall at 77.
- New AA-Paperclip benchmark tests 1,500 documents with all-or-nothing 15-answer scoring.
- Cost varies 60x across decision models due to per-question vs per-document input billing.
- Full results at artificialanalysis.ai.
Artificial Analysis has released Decision Index v2, rebuilding its leaderboard for decision models. These systems choose from a fixed set of options and assign each one a probability in a single API call. The update introduces a confidence-sensitive headline score, increases the share of private evaluation data, and changes the model rankings.
Confidence changes the ranking
The new Decision Score calculates each question as 50 × (1 + c − (p − c)2), where p is the model’s confidence in its selected answer and c equals 1 for a correct answer or 0 for an incorrect one.
- Correct answer: The score rises from 50 at zero confidence to 100 at full confidence.
- Incorrect answer: The score falls from 50 at zero confidence to 0 at full confidence.
Across an evaluation, the formula becomes 50 × (1 + accuracy − Brier score). The Brier score measures the mean squared gap between confidence and observed correctness, so lower values indicate better calibration. A model gains from accurate answers and loses ground when it assigns excessive confidence to errors.
Production routing systems commonly accept a model’s choice only above a confidence cutoff, sending lower-confidence cases to a person or a stronger model. Decision Score averages the payoff across every possible cutoff, allowing teams to compare models before selecting a threshold for their workload.
On the document benchmark, Clef records a 36% raw pass rate against 24% for Celeris-1.1 Decision. Celeris narrows the Decision Score gap with a Brier score of 0.15, compared with Clef’s 0.27. Across the full index, GLiDE v1.2 leads Drex Max v1.1 in accuracy, 60 to 58, while Drex ranks higher after calibration enters the calculation.
Six tests, nearly half private
Six evaluations contribute to the index, covering document processing, workplace judgment, tool use, factual reliability, reasoning, and prompt invariance.
| Evaluation | Weight | What it measures |
|---|---|---|
| AA-Paperclip | 30% | Decisions and extraction across 1,500 private resumes, receipts, invoices, and machine-learning papers. Each document includes five decisions and 10 facts, with all 15 answers required for a pass. |
| EvalSafe-O*NET-AA | 15% | Workplace judgment. |
| When2Call | 10% | Whether and when a model calls tools. Its weight fell from 25% in the first index. |
| AA-Omniscience-Decision v1.1 | 15% | Knowledge and hallucination. The evaluation rotates through 10 alternatives to the phrase “I don’t know.” |
| HLE-MultiChoice v1.1 | 15% | Reasoning with the same rotating abstention phrases, reducing reliance on phrase matching. |
| AA-Symmetry | 15% | Answer and probability stability across 120 base questions and 440 variants that reorder options, swap labels, or alter yes-or-no phrasing. |
Private items now account for 45% of the index, up from 25% in version one. The larger private share reduces opportunities to tune directly against known benchmark questions. WildGuardTest was removed after model performance approached its ceiling, leaving too little separation among competitors.
Three specialists bunch at the top
Three specialist decision models finish within one point of one another. Nace AI’s Drex Max v1.1 and Fastino Labs’ GLiDE v1.2 each score 73, followed by Base Labs’ Mammoth 27B at 72. The tested general-purpose language models score higher, led by Claude Haiku 5.5 (Max) at 77 and GPT-6 Luna (Max) at 76, with substantially higher costs and latency.
AA-Paperclip produces the widest performance gap. Claude Haiku 5.5 (Max) passes 92% of documents and GPT-6 Luna (Max) passes 86%. Drex Max v1.1 and GLiDE v1.2 lead the specialists at 63% and 62%.
The document-level requirement magnifies individual errors because one wrong answer causes the entire document to fail. Drex gets 97% of individual answers correct, close to Haiku’s 99%, while passing 63% of complete documents against Haiku’s 92%. As a rough illustration, a 97% per-answer success rate compounds to about 63% across 15 independent answers. Specialist models lose most of their points on document decisions, according to the evaluation, rather than fact extraction.
Repeated inputs drive the bill
Published costs among decision models span more than sixtyfold, from $14 per million tasks for Celeris-1.1 Decision to $859 for Clef. Provider token prices explain only part of that range because models process multi-question documents differently.
Drex Max v1.1 and Decider v1.2 bill a document’s input once for the full batch of questions. GLiDE v1.2 and Decider Flash v1.2 process each question separately, charging for the document input on every pass.
On AA-Paperclip, that execution pattern consumes roughly 63,000 input tokens per task for GLiDE v1.2 and 9,000 for Drex Max v1.1. GLiDE consequently costs about $780 per million tasks, compared with $102 for Drex, a difference of nearly eightfold despite list prices that are only 33% higher.
| Model | Index score | Cost per 1M tasks | Time per task |
|---|---|---|---|
| Claude Haiku 5.5 (Max) | 77 | $6,240 | 32.65 s |
| GPT-6 Luna (Max) | 76 | $1,530 | 11.34 s |
| Drex Max v1.1 | 73 | $102 | 275 ms |
| GLiDE v1.2 | 73 | $780 | Not listed |
| Mammoth 27B | 72 | Not listed | Not listed |
| Celeris-1.1 Decision | Not listed | $14 | 56 ms |
Abstention lifts symmetry scores
AA-Symmetry measures whether controlled prompt changes alter a model’s answer or confidence. Base Labs leads this evaluation, with Mammoth 27B scoring 96 and Elephant 9B scoring 93. Their results partly reflect frequent abstention and consistent reuse of that abstention across prompt variants.
Mammoth selects the same answer across variants on 99% of questions. Drex Max v1.1 reaches 91%, GLiDE v1.2 reaches 88%, and Clef reaches 67%. Clef therefore changes its selection on roughly one-third of reworded questions, which can alter automation and escalation decisions even when the underlying evidence remains constant.
Production tradeoffs favor specialists
GPT-6 Luna (Max) leads Drex Max v1.1 by three index points while costing about 15 times as much per million tasks and recording roughly 41 times the latency. Celeris-1.1 Decision completes a task in 56 milliseconds, while Mammoth 27B trails GPT-6 Luna by four points with roughly one-118th of its latency, according to the benchmark.
High-volume ticket classification, support routing, and structured-field extraction place cost and latency alongside quality. The relevant Pareto frontier contains models for which no available alternative is simultaneously higher-scoring, cheaper, and faster.
Production comparisons also need workload-specific checks beyond the aggregate index:
- Threshold behavior: Measure accepted-case accuracy and escalation volume at the confidence cutoff used in production.
- Request shape: Recalculate costs using the expected document length, question count, batching behavior, and repeated context.
- Failure policy: Test abstention handling, prompt variants, and document-level pass requirements against the application’s error budget.
Comments
