Artificial Analysis' MLCR-AA Shows Most AI Models Fail Medical Reasoning

Reading a medical claim file is not a search problem. It is a synthesis problem, where a human reviewer stitches together chronology, causality, and treatment patterns across hundreds of visits before writing a defensible summary. A new leaderboard from Artificial Analysis, built on top of Wisedocs' Medical Long Context Reasoning (MLCR) benchmark, tries to measure exactly that skill and shows just how far frontier models still have to go.

MLCR-AA evaluates a private held-out set of the hardest cases (expert-tier clinical synthesis and compound, multi-part reasoning), which is separate from the publicly released dataset. The Artificial Analysis implementation runs 60 questions drawn from those two tiers against complete case files, with each question repeated three times and averaged. Claude Fable 5 tops the board at 64.4%, but the median model scores under 15%, which tells you most of what you need to know about the current state of long-document medical reasoning.

What the benchmark actually asks

The underlying dataset comes from Wisedocs, a company that builds automation for insurance claims reviewers. They built 10 synthetic, real-world inspired medical cases that are between 25k and 64k tokens in length. These cases consist of 50-150 medical summaries spanning across specialities. Questions are organized into six tiers of difficulty, and MLCR-AA only runs the two hardest:

• Expert: the answer is not written anywhere in the record and must be reasoned out. They ask not just what happened, but why, and what it means. Example: why has the claimant not been able to return to work?
• Compound: two or more independent sub-questions in one prompt, testing whether a model can handle multiple asks without dropping or conflating them.

Three judges and a length gate

Grading is where MLCR-AA gets interesting. Every response first has to pass a conciseness check, which fails any answer longer than roughly five times the reference. Grading uses a three-model judge panel (Gemini 3.1 Pro, Claude Opus 4.8, and GPT-5.5) and aggregates decisions via majority vote for each task and criterion. Each answer is scored on two dimensions:

• Accuracy: is every fact the model states actually grounded in the source?
• Completeness: did the model include all the key facts an expert reviewer put in the reference answer?

A response only counts as correct if it is concise and wins the majority vote on both accuracy and completeness.

Accuracy is solved. Completeness is not.

The most useful finding is the split between the two grading axes. GPT-5.6 Terra (max) scores the highest on MLCR-AA Accuracy (Judged Responses) with a score of 93.7%, yet it lands only 10th overall because it omits too much. Meanwhile Claude Opus 5 (Adaptive Reasoning, Max Effort) scores the highest on MLCR-AA Completeness (Judged Responses) with a score of 86.1%. In plain terms: current models are largely right about what they choose to report, but they leave out roughly half of what a specialist would have written. For claims work, an answer that is accurate but incomplete can still get a claim wrong.

Anthropic's lead on the overall score is not because its accuracy is better than OpenAI's. It is because Claude models cover more of the expert reference answer per response.

Cost, open weights, and the Pareto picture

Top scores come at a steep price. Anthropic's leading configurations run between $0.30 and $1.00 per task, driven by heavy reasoning budgets on cases that can push 150 pages. Nova Lite scores the highest on MLCR-AA Conciseness with a score of 100.0%, though that is a length-only metric and does not indicate overall quality.

On the open-weights side, Kimi K3 (max) from Moonshot leads at 38.3% and does it at roughly a sixth of Claude's per-task cost, which puts it on the score-versus-cost Pareto frontier alongside GPT-5.6 Terra and Luna. If you are building a medical or claims pipeline where you plan to run millions of case files, that gap between 64.4% at $1 and 38.3% at $0.15 is exactly the trade-off you will end up modeling.

Why this benchmark matters

Long-context evaluation has largely been a needle-in-a-haystack story, where models get graded on retrieving one fact from a giant blob of tokens. MLCR-AA is closer to what a knowledge worker actually does: read 100 pages, understand the story, and write a defensible answer that a human expert would sign off on. The dataset and harness are open sourced on GitHub and Hugging Face for the first three difficulty tiers, so teams can reproduce the easier portions locally and only rely on the Artificial Analysis hold-out for the hard tiers.

The takeaway for anyone deploying LLMs against long professional documents: pick your model based on completeness, not just accuracy, and expect to pay real money to cover the full scope of an expert answer.