Vals AI's ClaimBench Shows Perplexity's Decider V1.2 Beats Finance Verification at $0.10

Vals AI has released ClaimBench, a finance benchmark that tests whether a model can verify that an SEC filing supports a specific claim. Perplexity’s Decider V1.2 leads the dedicated decision models with 99.69% accuracy while costing about $0.10 per 1,000 claims.

Citation checking often reduces to a fixed-label decision: the evidence supports a claim, contradicts it, or fails to address it. ClaimBench tests whether smaller classifiers can handle that work cheaply and route uncertain cases to a frontier language model.

Filings engineered to expose mistakes

ClaimBench contains 1,960 claims, each paired with an excerpt from a Form 10-K. The system assigns one of three labels: supported, contradicted, or unsupported. Vals AI derives the gold labels through deterministic transformations, making their provenance reproducible and removing variance from model-based judging.

The benchmark divides the claims into five categories of roughly 392 examples each:

• Direct support: A paraphrase of a sentence in the excerpt.
• Subtle contradiction: A small edit that reverses the meaning, such as changing “less established” to “more established.”
• Numeric discrepancy: An amount or percentage replaced with a plausible but incorrect value.
• Missing evidence: A sentence taken from another passage in the same filing.
• Non-supporting citation: A claim about one company paired with similar language from another company’s filing.

Small models crowd frontier accuracy

A decision model returns probabilities over predetermined labels in one inference pass. Its fixed output space removes the token cost and parsing work associated with free-form generation, making it suitable for high-volume classification.

Four frontier language models achieved 100% accuracy: Claude Sonnet 5.5, Claude Haiku 5.5, Claude Opus 5.5, and GPT-6.1 Sol. The dedicated models finished within 1.33 percentage points of that result.

These percentages describe a fixed dataset, while the cost estimates reflect its prompt lengths and the providers’ prices at measurement time. Production costs will vary with document length, batching, provider, and hosting method.

Confidence routing cuts the bill

Vals AI also evaluated a cascade that accepts Decider V1.2’s answer when its highest probability clears a calibrated threshold. Claims below the threshold move to Claude Haiku 5.5, giving the pipeline a defined escalation rule and measurable error budget.

On the held-out split, Decider V1.2 handled 99.7% of claims and escalated the remaining 0.3%. The combined system reached 99.85% accuracy at $0.11 per 1,000 claims. Haiku alone reached 100% for $1.19, so the cascade reduced the reported cost by about 91% while giving up 0.15 percentage points of accuracy.

Reported latency followed the same pattern: Decider V1.2 took 1.23 seconds per claim, Claude Haiku 5.5 took 5.06 seconds, and Claude Opus 5.5 took 6.47 seconds. At document-review scale, throughput and concurrency limits can amplify those differences.

One altered phrase still causes trouble

Subtle contradictions produced the lowest scores because the claim and evidence retain high lexical overlap while changing a relation, direction, or qualifier. Decider V1.2 scored 98.5% in this category, followed by Mercury Decide at 97.7% and Liquid AI d1 at 94.6%.

Performance declined more sharply for weaker models. Cloudflare’s Clef Omni scored 55.9% on subtle contradictions, and ScaleDown Classify-1 reached 66.8%. The leading models made fewer errors on numeric discrepancies and missing evidence.

Inside Decider V1.2

pplx-decider-v1.2 is Perplexity’s 27-billion-parameter multimodal decision model, built on the Qwen3.8-27B backbone according to the company. It accepts a shared input, called the state, as text, JSON, or images and returns probability distributions for 1 to 128 named questions.

Each question can use one of three output formats:

• Boolean: A yes-or-no decision.
• Choice: Selection among as many as 255 options.
• Score: Selection among as many as 10 ordered levels.

Version 1.2 processes the state and every question in one sequence. Each question receives a separate readout at a marker token placed after its options. Perplexity says this joint, single-pass design lets several questions about the same state cost roughly as much as one and allows related questions to share context.

Joint processing can reduce accuracy when independent candidates compete for the same judgment. Perplexity reported a 1.1-point decline on function-calling benchmarks, where the model may need to evaluate several candidate tools separately.

Memory depends on precision and context

Perplexity published the 27B model and the 4B pplx-decider-flash-v1.2 on Hugging Face under the Apache 2.0 license. Both are also available through hosted APIs, with the Flash variant targeting lower cost and smaller hardware.

The reported 28 to 29GB footprint for the 27B model implies an 8-bit, FP8, or similarly compact weight representation. BF16 weights alone require roughly 54GB before accounting for activations, the key-value cache, and runtime overhead.

A 32GB GPU may therefore load a reduced-precision version at shorter context lengths. Using the model’s maximum 256K-token context requires additional memory, so deployment sizing must account for sequence length, batch size, cache representation, and inference framework.

Calling the Decisions API

The Vercel AI SDK exposes the structured interface through experimental_decide. A minimal request through the Vercel AI Gateway looks like this:

import { experimental_decide as decide } from 'ai';

const result = await decide({
  model: 'perplexity/pplx-decider-v1.2',
  state: 'The support agent issued a full refund to the customer.',
  questions: {
    refunded: {
      type: 'boolean',
      instructions: 'Was a refund issued?',
    },
  },
});

The state contains the evidence shared across questions, while each question defines its label space and instructions. The route uses the Decisions API schema supported by OpenRouter and the Vercel AI SDK. Existing chat-completions clients need an adapter because the request and response structures differ.

What ClaimBench leaves open

ClaimBench measures a narrow, useful task with deterministically constructed examples. Its results leave several production conditions untested:

• Ambiguous claims that require legal or accounting judgment.
• Evidence spread across several passages or filings.
• Tables, charts, OCR errors, and malformed documents.
• Retrieval failures that prevent the verifier from seeing the relevant passage.
• Domain shifts outside SEC filings.
• Confidence calibration after model, prompt, or data changes.

A production evaluation should therefore include representative documents, naturally occurring errors, adversarial edits, and separate calibration data. Teams also need to audit a sample of high-confidence decisions, since a confidence threshold controls observed risk only while the deployment distribution resembles the calibration set.

A practical pattern for fixed-label work

ClaimBench supports a production architecture for citation checking, compliance review, ticket routing, and tool selection: run a low-cost decision model first, accept calibrated high-confidence outputs, escalate uncertain cases, and monitor both paths for drift.

The pattern works best when labels are explicit, mistakes can be measured, and a stronger fallback model is available. Under those conditions, the cascade converts model confidence into a routing decision and reserves expensive inference for the small share of cases that need it.