Decision models near LLM scores with lower latency and cost

Artificial Analysis has launched the Decision Index v1.0, a benchmark for models that choose among fixed options and assign each option a probability. Its leading decision model scores three points behind Claude Haiku 5.5 (Low), yet completes a task about 11 times faster and costs roughly 4.4 times less in the published comparison.

Decision models accept a state plus one or more fixed-choice questions, then return a selected option and probabilities in a single API call. Their constrained output suits routing, classification, safety filtering, and tool selection. The probabilities also allow developers to test calibration, which measures whether confidence scores match observed accuracy.

Four tests set the score

The Decision Index gives equal weight to four evaluations, with each contributing 25 percent of the composite score. The mix covers knowledge, reasoning, tool use, and classification so that one capability cannot dominate the result.

• AA-Omniscience-Decision: Uses 500 private questions with four candidate answers, exactly one correct answer, and an “I don’t know” option. Accuracy and non-hallucination rate contribute 12.5 percent each.
• HLE-MultiChoice: Uses 513 text-only multiple-choice questions from Humanity’s Last Exam. Answers are scored by exact letter match.
• When2Call: Presents conversations in which an assistant can use tools. The model must answer directly, call a tool, request more information, or report that it cannot answer.
• WildGuardTest: Asks up to three yes-or-no questions about harmful requests, refusals, and harmful replies. It uses average F1, a metric that balances precision and recall.

Jev leads the specialists

Jev 1.13 from TypeSafe leads the dedicated decision models with a score of 58. GLiDE from Fastino and Microsoft-Decision-1 follow at 54.

Perplexity’s Decider v1.1 27B records the lowest reported cost at $17 per million tasks. Microsoft-Decision-1 costs $31 and Jev 1.13 costs $42 per million tasks. Artificial Analysis places all three on the capability-cost Pareto frontier, meaning no measured alternative is both cheaper and higher-scoring.

Celeris-1 Decision posts the lowest reported latency at 0.06 seconds per task, or 60 milliseconds. Jev 1.13 takes 0.13 seconds. Claude Haiku 5.5 (Low) reaches 61 points, compared with Jev’s 58, but requires 1.4 seconds per task and costs $184 per million tasks.

The gap depends on the job

Leading decision models perform comparably to LLMs on classification and tool calling. LLMs retain a substantial advantage on broad knowledge and multi-step reasoning, with the clearest gaps appearing in AA-Omniscience-Decision and HLE-MultiChoice.

That task-level split limits where a dedicated endpoint can replace an LLM call. Stable labels and narrow actions align with a decision model’s fixed output space. Tasks that require synthesis, explanation, or reasoning across several steps remain better suited to an LLM.

Confidence runs hot

Across almost every confidence bucket, the tested decision models achieve lower accuracy than their stated probabilities imply. A well-calibrated model that reports 80 percent confidence should be correct on roughly 80 percent of comparable predictions. These models fall short of that standard, especially on knowledge and reasoning tasks.

WildGuardTest and When2Call show better calibration than the knowledge and reasoning evaluations. Production systems that route requests or trigger fallbacks at fixed confidence thresholds should therefore validate those thresholds on representative data and recalibrate the probabilities when necessary.

Same task, different API contract

Each benchmark item contains a state, such as a conversation or text sample, followed by one or more questions. A question either names several possible choices or requests a yes-or-no decision. For binary questions, the benchmark selects “yes” when its probability reaches at least 0.5.

Decision APIs return the chosen option and a probability for every option. LLMs receive equivalent task content through structured outputs. This setup compares the decisions produced by each system while preserving the interface each model is designed to use.

A practical deployment split

• Use a decision model for high-volume routing, intent classification, safety checks, and tool-call gating when the option set is stable.
• Keep an LLM for tasks that depend on broad knowledge, multi-step reasoning, synthesis, or generated explanations.
• Validate confidence thresholds on production-like traffic before using probabilities to approve actions or invoke fallbacks.
• Measure the full workload because label distribution, ambiguity, and domain language can shift accuracy, latency, and cost from benchmark results.

Artificial Analysis publishes per-benchmark results with the index and documents task construction, scoring, and model handling in its methodology. Teams evaluating these endpoints can use the benchmark as a shortlist, then test the leading candidates against their own labels, traffic, and error costs.