Artificial Analysis' HealthBench Rankings Show Verbose AI Medical Answers Lose Big

Artificial Analysis has published independent leaderboards for OpenAI’s HealthBench and HealthBench Professional. The results compare how leading language models handle open-ended health conversations, including their accuracy, communication, resilience to adversarial prompts, response length, and reported cost per task.

Medical multiple-choice benchmarks increasingly struggle to separate top models. HealthBench evaluates behavior across multi-turn conversations, which more closely resembles patient chat, clinical documentation, research support, and care consultation.

Two benchmarks, two audiences

HealthBench Professional uses fewer tasks but concentrates on work that demands clinical expertise. The broader HealthBench includes more consumer-facing exchanges and a wider range of communication demands.

A judge model sets the score

Each response is evaluated against a conversation-specific rubric whose criteria carry weights from minus 10 to plus 10 points. A judge model reads the full transcript and determines which criteria the response satisfies. Artificial Analysis uses GPT-5.4 at its low setting for HealthBench Professional and GPT-4.1 for HealthBench, matching OpenAI’s original evaluation setup.

The leaderboards report two scores for each evaluated configuration:

• Raw score: the share of available positive rubric points earned.
• Length-adjusted score: the raw result after a character-based correction rewards concise answers and penalizes long ones.

HealthBench Professional adjusts the score by 1.47 percentage points for every 500 characters above or below a 2,000-character baseline. HealthBench uses a steeper adjustment of 2.99 points per 500 characters. The correction is applied to individual responses before the benchmark average is calculated.

A 6,000-character answer loses about 11.8 points on HealthBench Professional solely from the length correction. That penalty can move a model several places even when its response satisfies more clinical criteria.

OpenAI fills the clinical cost frontier

On HealthBench Professional, GPT-6 Astra at its maximum setting leads with a 66.0% length-adjusted score. GPT-6.1 Sol follows at 65.1%, and Claude Sonnet 5.5 with maximum reasoning and fallback reaches 63.0%. The parenthetical labels identify the configurations tested by Artificial Analysis, so the results may differ from default API settings.

OpenAI models occupy the benchmark’s full Pareto frontier for score and reported cost per task. Each frontier model offers a score or cost advantage that the other evaluated configurations do not simultaneously match.

GPT-6 Luna trails Astra by 4.1 points while costing less than 2% as much per task. High-volume systems that can accept that score difference may find Luna a more practical starting point for internal evaluation.

Verbosity rewrites the order

Claude Opus 5.5 and Claude Fable 5.1 average about 5,900 characters per HealthBench Professional response, nearly three times the adjustment threshold. Each loses more than 11 points after the length correction.

Claude Opus 5.5 with maximum reasoning and fallback earns the benchmark’s highest raw score at 71.4%, then falls to sixth place with a 59.9% adjusted score. GPT-6 Sol is the only evaluated model averaging fewer than 2,000 characters on this benchmark, at roughly 1,600, and the only model whose score rises after adjustment.

The broader HealthBench produces a similar shift for Gemini 4 Argon at its high setting. Its 60.0% raw score ranks sixth, but responses averaging about 3,800 characters reduce its adjusted score to 49.1% and move it to 13th place.

The broader benchmark favors Claude

Claude Sonnet 5.5 with maximum reasoning and fallback leads the general HealthBench at 62.1% length-adjusted. DeepSeek V4.1 Flash at its maximum setting follows at 60.9%, and GPT-6.1 Sol reaches 58.5%.

The cost frontier is more diverse on this benchmark. GPT-6 Luna costs a reported $0.002 per task, DeepSeek V4.1 Flash costs $0.005, and Claude Sonnet 5.5 costs $0.08. Sonnet’s 1.2-point advantage over DeepSeek comes at roughly 16 times the per-task cost.

Completeness and clarity remain split

Two of HealthBench’s five tagged evaluation axes show the central response-design tradeoff:

• Completeness: whether the response includes the information required by the rubric.
• Communication quality: whether the response is clear, well structured, and appropriate in length and detail.

On the broader HealthBench, GPT-6 Sol produces the shortest answers at roughly 1,000 characters and leads communication quality with an 81% raw score. It also records the lowest completeness score at 34%. Claude Sonnet 5.5 leads completeness at 63%.

No evaluated model combines the strongest completeness and communication scores. Product teams therefore need to inspect axis-level results and sample outputs, particularly when missing a warning or follow-up question carries more risk than exceeding a display-length target.

Adversarial prompts widen the gaps

HealthBench Professional’s red-team conversations use adversarial prompts designed to expose unsafe, brittle, or easily manipulated behavior. Scores on these cases range from 30% for DeepSeek V4.1 Flash to 65% for GPT-6 Astra, compared with a range of 63% to 79% on good-faith conversations.

Claude Fable 5.1 with maximum reasoning and fallback leads good-faith cases at 79% but falls to 53% under red teaming. Claude Haiku 5.5 leads the writing-task category at 77%, showing that aggregate rank can conceal substantial differences among task types.

How developers can use the results

1. Consumer health chat: DeepSeek V4.1 Flash is a cost-efficient candidate for further testing, finishing 1.2 points behind Claude Sonnet 5.5 on the broader benchmark at a fraction of the reported cost.
2. Clinician-facing workflows: GPT-6 Astra and GPT-6.1 Sol lead the professional benchmark, with Astra also posting the strongest reported adversarial score.
3. Length-constrained interfaces: raw and adjusted score gaps identify models that may require stricter prompts, output limits, or post-processing for mobile, voice, and triage-summary applications.
4. Model routing: axis and task-level scores can support routing by use case, such as assigning concise models to summaries and more complete models to draft documentation.

Limits before deployment

Several constraints limit how far teams should extend these results beyond model screening:

• Model-based grading: rankings can change with the judge model, rubric interpretation, and evaluation prompt. Both reported judges are OpenAI models.
• Benchmark provenance: Artificial Analysis runs the independent leaderboard, while OpenAI created the benchmarks and original evaluation setup.
• Statistical uncertainty: small score differences may lack practical or statistical significance, especially on the 525-task professional set.
• Cost variability: per-task figures depend on the tested configuration, response length, and current API pricing.
• Clinical validity: benchmark performance does not establish safety, effectiveness, regulatory compliance, or improved patient outcomes.

A high HealthBench score supports model selection within a standardized harness. Production deployment still requires use-case-specific testing, clinician review, safety monitoring, privacy controls, regulatory analysis, and prospective evaluation with representative users.