Artificial Analysis Finds 1 in 4 AI Endpoints Quietly Underperform Their Advertised Models

Artificial Analysis has released version 1.1 of its Endpoint Accuracy Index, which compares hosted open-weight models with reference deployments of their official weights. Across more than 100 serverless endpoints, roughly one in four scores below its reference deployment.

That gap matters because a model name does not fully describe the product behind an API. Providers may quantize weights, use custom kernels, modify inference settings, limit output length, or introduce implementation bugs. The index measures how much accuracy each endpoint preserves, adding a quality signal to the usual comparisons of price, latency, and throughput.

What a 100% Score Means

Artificial Analysis reruns each endpoint against three equally weighted evaluations, then compares the results with a self-hosted baseline using the official weights at the recommended precision. A score of 100% matches the reference result. Scores above 100% can occur because repeated evaluations still contain sampling variance, so the accompanying 95% confidence interval remains essential.

Reference configurations are documented down to the serving software and hardware. The GLM-5.3 reference, for example, uses eight NVIDIA B200 GPUs, SGLang 0.5.18, native FP8 weights with DeepGEMM kernels, EAGLE speculative decoding, and a 262,000-token context window. Tests use each hosted endpoint’s highest supported reasoning mode, enable streaming, and request its maximum output allowance.

Version 1.1 Narrows the Noise

• AA-LCR moves to version 1.1. The release corrects two answer keys and introduces an updated equality-checking model for scoring.
• Long-context repeats increase from 10 to 25. The average 95% confidence interval falls from plus or minus 4.7 points to plus or minus 2.6 points, reducing uncertainty in close endpoint comparisons.
• Coverage expands to six models. The index now includes DeepSeek V4.1 Flash 1017, GLM-5.3, GLM-5.3 Flash, Muse Spark Open, Kimi K3, and gpt-oss-120b.

GLM-5.3 Flash Exposes Silent Failures

The GLM-5.3 Flash results cover listings from 24 providers, including Together AI, Fireworks, Baseten, DeepInfra, CoreWeave, and Z.ai, with scores available for 22 endpoints. Sixteen scored endpoints are statistically consistent with the reference, while one reaches 104.2%. The weakest results trace to two operational choices:

• Output limits truncate answers. Two endpoints with output caps near 32,000 tokens cut off about half their HLE-250 responses before producing a final answer. Their raw HLE-250 accuracy falls to 22% and 29%, compared with 55% for the reference.
• Shorter reasoning reduces accuracy. Two other endpoints generate roughly 30% fewer reasoning tokens on HLE-250 without recorded truncation. Their raw accuracy reaches 48%, seven points below the reference.

Provider economics vary just as sharply. Blended cost per Intelligence Index task spans a 4.4-fold range, from $0.41 at GMI to $1.81 at Modal. Completion time ranges from 5.5 minutes at Inco’s FAST endpoint to 51.5 minutes at SiliconFlow. An endpoint scoring 79% or 83% may therefore look competitive on price or speed while delivering materially weaker results.

A Schema Bug Sinks Tool Use

Meta’s Muse Spark Open adds a narrower example of endpoint-specific failure. Three of five endpoints tested at xhigh reasoning score between 99.5% and 99.9% of the reference, including one deployment using FP4 weights. The other two encode tool arguments declared as strings, such as booking IDs, as numbers.

That schema mismatch lowers their raw BFCL v4-500 scores to 50% and 51%, compared with 59% for the reference. The same endpoints remain within two points of the baseline on HLE-250 and AA-LCR-25, so workloads without tool calling may never encounter the defect.

Output Limits Drive the Largest GLM Gaps

GLM-5.3 has the index’s broadest coverage, with 26 endpoints scoring from 67.2% to 99.3%. Its three largest deficits stem from output caps and reasoning loops that truncate HLE-250 responses. Five of the six FP4 endpoints remain statistically consistent with the FP8 reference, weakening the assumption that lower-precision weights always impose the largest accuracy cost.

DeepSeek’s provider results show the tightest range in the index. All 22 endpoints are statistically consistent with the reference, with scores spanning 97.5% to 101.4%.

A Practical Provider Check

Teams selecting a hosted open-weight model can use the index as one stage of a broader endpoint evaluation:

1. Shortlist providers whose price, latency, throughput, context window, and regional availability fit the workload.
2. Check the accuracy score and 95% confidence interval for the exact model and endpoint.
3. Inspect the evaluation breakdown. BFCL matters most for tool-using agents, HLE for difficult reasoning, and AA-LCR for long-context retrieval.
4. Review notes about output caps, truncation rates, reasoning settings, and schema bugs.
5. Run application-specific canaries before deployment, especially for structured output, multilingual prompts, and domain-specific tasks outside the index.

The index covers a fixed evaluation suite, and hosted configurations can change after measurement. Even within that scope, the results show that a checkpoint name is an incomplete API specification. Documented reference deployments and endpoint-level confidence intervals make quality loss visible before it reaches production.