Artificial Analysis Exposes How API Providers Quietly Degrade Model Accuracy
Artificial Analysis launches the Endpoint Accuracy Index, revealing how much model intelligence providers silently trade away for speed and cost savings
- New index launched: Artificial Analysis's Endpoint Accuracy Index measures how much of a model's accuracy each API provider preserves vs. a self-hosted reference.
- Three benchmarks: Tool calling (BFCL v4-500), hard reasoning (HLE-250), and long-context recall (AA-LCR-25), equally weighted at 33% each.
- GLM-5.2 finding: The most restrictive provider endpoints score half the reference or less on hard reasoning due to output token limits cutting off reasoning mid-chain.
- gpt-oss-120b finding: Tool call handling varies wildly -- some endpoints score 22% on BFCL-500 vs. 37% for the reference deployment.
- DeepSeek V4 Pro finding: Most endpoints are at reference parity; DeepSeek's own API scores slightly above the reference.
- Coverage is live for GLM-5.2, gpt-oss-120b, and DeepSeek V4 Pro; Kimi K3 coming soon.
When you call a model API, you assume you're getting the model. You're often not. Providers quantize weights to lower precision, cap output token limits, tune sampling defaults, and sometimes just ship bugs. The result is that the same model can behave very differently depending on who's serving it. Artificial Analysis just launched a new index to put a number on exactly how much accuracy each provider endpoint preserves.
What is the Endpoint Accuracy Index?
The Endpoint Accuracy Index measures how the intelligence and capability of a specific model varies across API providers who serve it. The same model served by different providers can behave differently depending on quantization, sampling defaults, context handling, token limits, prompt parsers, and other endpoint-side configuration.
Artificial Analysis self-hosts each model as a reference deployment, expressing each provider endpoint's accuracy score as a percentage of that baseline. 100% means the endpoint matches the reference; lower scores indicate accuracy loss relative to the baseline.
Each result carries a 95% confidence interval computed from per-repeat variation, and endpoints are flagged as either statistically within the reference or significantly below it.