Artificial Analysis Finds Claude Sonnet 5.5 and GPT-6 Astra Fail Differently

Artificial Analysis has refreshed its MLCR-AA leaderboard, which tests whether frontier models can read, chronologically reconstruct, and reason over synthetic patient records spanning 25,000 to 64,000 tokens. Claude Sonnet 5.5 and GPT-6 Astra share first place at 41.7%, followed by GPT-6.1 Sol at 38.3%. Qwen3.8 2.4T A95B leads models with released weights at 30.6%.

The tied score conceals distinct failure patterns that affect model selection for clinical document review. Claude configurations lead on completeness, while OpenAI configurations produce fewer claims that contradict the record. Those errors require different safeguards in claims review, chart summarization, evidence extraction, and clinical research workflows.

A tie with opposite failure modes

MLCR-AA awards an overall pass when an answer includes every key point from the expert reference and contains no claim that contradicts the record. Completeness and accuracy are also reported separately, revealing how models can reach the same overall score through different combinations of coverage and fidelity.

Each result is specific to the listed inference configuration, including settings such as “max” and “max with fallback.” Across all tested configurations and effort levels, Anthropic models hold the top 18 completeness positions. OpenAI models hold the top 27 accuracy positions. In practice, Claude answers more often cover every reference point, while OpenAI answers more often remain free of record contradictions.

A new pipeline resets the baseline

The refreshed scores belong to a new evaluation series because Artificial Analysis changed the prompts, answer format, references, and judging process to resemble medical record review more closely. Earlier MLCR-AA scores therefore provide no direct trend line for the current leaderboard.

• Role and prompt order: Models receive a system prompt that frames them as medical record reviewers, and the question appears after the complete record.
• Answer structure: Each requested part receives one paragraph containing a conclusion and supporting evidence. The standardized format replaces the previous concision gate.
• Judge panel: GPT-6.1 Sol, Claude Opus 5.5, and Grok 4.7 grade responses at medium effort. Each judge identifies contradictions and missing reference points.
• Pass rule: A response passes when a majority of judges find no contradiction and no missing key point.
• Repeated sampling: Every question runs three times, and the leaderboard reports the mean result.
• Reference audit: Reviewers checked all 60 expert answers against the source records and refined 15 of them.

The revised panel reaches a unanimous verdict on 69% of answers, compared with 48% for the previous panel. MLCR-AA remains separate from the Artificial Analysis Intelligence Index, so its results do not affect that broader ranking.

Private questions raise the bar

Wisedocs created Medical Long Context Reasoning, or MLCR, from 10 synthetic cases containing 50 to 150 visits across multiple specialties. Its six difficulty tiers progress from retrieving a single fact to synthesizing clinical evidence and answering compound, multi-part questions across a longitudinal record.

MLCR-AA draws from a private, held-out set covering the two hardest tiers, which reduces the risk that benchmark questions appeared in model training data. The harness can also insert OCR-processed pages from real U.S. and Canadian medical forms as filler, adding plausible distractions around the relevant evidence.

Developers can inspect the public cases and tier 1 through tier 3 questions through the Wisedocs benchmark and Hugging Face. The GitHub harness provides the evaluation code.

Cheap models can spend more tokens

Reported per-task costs span more than 200-fold across the selected configurations, making score, token use, and price separate procurement variables for high-volume record review.

Reported output-token use also varies sharply. Claude Haiku 5.5 produces about 67,000 output tokens per task, roughly twice Sonnet 5.5 at 34,000 and about 20 times GPT-6 Astra at 3,500. Haiku’s lower token price keeps its task cost near $0.04, and its 35.0% score exceeds Opus 5.5 at a fraction of the reported cost. GPT-6 Luna needs a reported quality score before a score-to-cost comparison is possible.

Model choice follows the failure budget

The benchmark supports different choices for different workflow risks, with each option requiring controls for its weaker axis.

The highest observed pass rate remains 41.7%, meaning the leading configurations fail most questions under the benchmark’s strict combined criterion. The median model fails more than 85% of them. Production use in clinical or claims settings therefore requires evaluation on representative records, explicit omission and contradiction checks, privacy and security review, monitoring, and qualified human oversight.