Skip to main content
Back

MedCUA-Bench Reveals Top AI Agents Collapse to 9% on Real Hospital Software

A new benchmark tests AI agents on real clinical software -- and the results reveal a dangerous gap between lab performance and hospital-ready reliability

  • New benchmark: MedCUA-Bench tests AI computer-use agents on 18 clinical scenarios across 10 medical domains using real-style GUIs.
  • Alarming performance gap: Best closed-source model hits 54.2% on synthetic UIs but all models fall below 9% on real OpenEMR software.
  • Open-source agents lag badly: Open-source agents average only 2.5% success; the best reaches 16.2%.
  • Safety-first evaluation: Each task is scored across five clinical safety dimensions, not just task completion, catching dangerous errors.
  • Novel dual-goal design: Tasks have both intent-level and step-level goals to separate clinical reasoning failures from UI execution failures.
  • Reproducible infrastructure: Fully self-contained HTML environments run via Playwright -- no hospital system access or licensing required.

AI agents that can click through software interfaces and complete tasks autonomously are one of the most hyped capabilities in the field right now. OpenAI has Operator, Anthropic has Claude Computer Use, and a growing ecosystem of open-source frameworks is racing to catch up. The pitch for healthcare is obvious: clinicians spend enormous amounts of time on repetitive screen-based work -- entering orders, updating records, navigating complex EHR interfaces. But how well do these agents actually perform on real clinical software? A new benchmark called MedCUA-Bench has a sobering answer.

The problem no one was measuring

Most computer-use agent (CUA) benchmarks -- tools that measure how well an AI can control a computer by looking at screenshots and taking actions -- test on general web tasks or consumer desktop software. Prior benchmarks for interactive healthcare agents have focused mainly on clinical workflows within single, unified EHR environments. And even those that do touch healthcare, like MedAgentBench, challenge AI agents with clinically relevant tasks that require interactions with a FHIR-compliant environment -- meaning they work through structured data APIs, not actual graphical interfaces.

That gap matters enormously in practice. Real clinical software looks nothing like a consumer web app. Medical GUIs are dense, use domain-specific terminology, and require the agent to understand clinical context to avoid dangerous mistakes -- not just complete a task. No standardized benchmark exists for evaluating the agent capabilities of LLMs in medical contexts, which have unique intricacies and highly specialized data. Robust evaluation is essential for safe AI deployment, but the lack of benchmark datasets hinders agent adoption in healthcare due to a lack of trust, safety concerns, and regulatory hurdles.