Artificial Analysis Exposes How API Providers Quietly Degrade Model Accuracy
Artificial Analysis launches the Endpoint Accuracy Index, revealing how much model intelligence providers silently trade away for speed and cost savings
- New index launched: Artificial Analysis's Endpoint Accuracy Index measures how much of a model's accuracy each API provider preserves vs. a self-hosted reference.
- Three benchmarks: Tool calling (BFCL v4-500), hard reasoning (HLE-250), and long-context recall (AA-LCR-25), equally weighted at 33% each.
- GLM-5.2 finding: The most restrictive provider endpoints score half the reference or less on hard reasoning due to output token limits cutting off reasoning mid-chain.
- gpt-oss-120b finding: Tool call handling varies wildly -- some endpoints score 22% on BFCL-500 vs. 37% for the reference deployment.
- DeepSeek V4 Pro finding: Most endpoints are at reference parity; DeepSeek's own API scores slightly above the reference.
- Coverage is live for GLM-5.2, gpt-oss-120b, and DeepSeek V4 Pro; Kimi K3 coming soon.
When you call a model API, you assume you're getting the model. Often you are not. Providers quantize weights to lower precision, cap output token limits, tune sampling defaults, and sometimes ship bugs. The same model can behave very differently depending on who serves it. Artificial Analysis just launched an index that puts a number on exactly how much accuracy each provider endpoint preserves.
What the Index Measures
The Endpoint Accuracy Index measures how a model's intelligence and capability vary across the API providers serving it. Quantization, sampling defaults, context handling, token limits, and prompt parsing all introduce differences between what the model lab ships and what you actually call.
Artificial Analysis self-hosts each model as a reference deployment, then expresses each provider endpoint's score as a percentage of that baseline. A score of 100% means the endpoint matches the reference; anything lower reflects accuracy lost to endpoint-side configuration. Each result carries a 95% confidence interval computed from per-repeat variation, and endpoints are flagged as either statistically within the reference or significantly below it.