MentalHealthBench tests AI beyond crisis responses

Many public mental-health evaluations focus on whether an AI model responds safely during an immediate crisis. That emphasis leaves out much of what appears in chat products: everyday stress, relationship conflict, caregiving questions, and ambiguous distress. OpenAI’s MentalHealthBench evaluates model responses across that broader range.

Coverage from daily stress to emergencies

OpenAI co-created the benchmark with more than 80 licensed psychologists and psychiatrists from 22 countries. The group spoke 19 languages and represented nearly 20 mental-health subspecialties.

The benchmark covers adults, teenagers, caregivers, and clinicians across multiple languages, regions, and topics. Its scenarios fall into three acuity levels:

• Non-acute: Everyday conversations involving stress or emotional strain.
• High-acuity: Serious concerns or significant distress without an immediate emergency.
• Emergencies: Immediate safety concerns that require real-world support.

The team used privacy-preserving methods to create synthetic conversations that approximate real patterns of mental-health use without publishing private user exchanges.

Rubrics built conversation by conversation

Each synthetic conversation receives an expert-written rubric for evaluating the model’s reply to the final user message. The grading process has three main steps:

1. Experts define specific criteria. Each criterion evaluates one behavior, such as asking a relevant question, acknowledging distress, or offering appropriate guidance.
2. Criteria receive clinical weights. Scores range from -10 to +10. Positive values reward beneficial behavior, negative values penalize harmful behavior, and larger absolute values indicate greater clinical importance.
3. Consensus filters the rubric. At least three experts review each conversation. A criterion remains only when at least two agree and no reviewer contradicts it.

An automated grader compares model responses with the approved criteria, allowing OpenAI and external researchers to score many outputs consistently.

Ten signals beneath the aggregate

MentalHealthBench reports an overall result alongside ten dimensions of model behavior. The release highlights capabilities such as seeking relevant context, preserving user agency, and providing actionable guidance. Two models with similar totals can therefore have different product risks.

Behavior-level results can help teams locate failures that an aggregate score conceals. A model might provide practical advice while making unsupported assumptions, or respond warmly without collecting enough context to interpret an ambiguous situation.

OpenAI reports that newer frontier models perform better on the evaluation, particularly when deciding when to seek more context. Model-level results appear in the report’s charts instead of a single leaderboard table.

Users and clinicians reward different qualities

A companion study compared expert rubrics with feedback from 44 adults who had used AI for mental-health or emotional support. Participants represented 16 countries and spoke 14 languages.

Users placed greater weight on tone and practical next steps. Clinicians emphasized gathering relevant context and interpreting ambiguous statements carefully. The benchmark’s final scoring follows expert consensus, so its scores reflect clinical priorities more directly than user preferences.

That difference gives product teams two distinct evaluation targets. Clinical rubrics can test safety and judgment, while user research can assess whether responses are understandable, respectful, and useful in practice.

What product teams can test

Mental-health conversations can emerge in general-purpose chat products even when emotional support is not the intended use case. The benchmark gives developers several components they can apply before release:

• Reusable evaluation design: Teams can adapt the weighted, conversation-specific rubric method used in MentalHealthBench and HealthBench.
• Behavior-level diagnostics: Scores for context seeking, agency, and actionable guidance can inform system prompts, model selection, tool routing, and escalation policies.
• Teen scenarios: A dedicated persona track reviewed by youth mental-health clinicians can support testing for products accessible to users under 18.
• Repeatable comparisons: Developers can run the same scenarios against candidate models, prompts, and safety configurations to identify regressions.

Boundaries of the benchmark

MentalHealthBench measures responses to defined synthetic scenarios. It does not establish that a model is clinically safe, suitable for therapy, or reliable across long-running conversations. Synthetic data may also omit details and interaction patterns found in real use.

Automated grading introduces another source of error, especially for nuanced or borderline responses. Production evaluations should pair benchmark scores with clinician review, adversarial testing, privacy controls, age-appropriate safeguards, and procedures for directing urgent cases to real-world help.

The research paper documents the dataset construction, grading method, and model results. OpenAI has released the benchmark for external evaluation, giving researchers and developers a shared method for testing mental-health responses while preserving the need for clinical oversight.