UC Santa Cruz and NVIDIA's AutoMedBench Reveals Why AI Agents Fail Medical Research
AutoMedBench reveals that frontier AI agents fail medical research pipelines not from lack of knowledge, but from poor validation and error recovery
Takeaways
- AutoMedBench is a new benchmark from UC Santa Cruz and NVIDIA that evaluates AI agents on full medical research workflows, not just final predictions. [Paper]
- Agents are scored across a 5-stage pipeline (Plan, Setup, Validate, Inference, Submit) over 24 tasks spanning segmentation, VQA, report generation, image enhancement, and lesion detection.
- Claude Opus 4.6 leads the leaderboard at 66.5 overall, but costs $19.77/run; GLM-5 scores 61.6 at only $2.73/run, making it the strongest cost-efficient option.
- Validation (S3) is the weakest stage across all agents; verification and submission errors account for 75.8% of all tagged failures, while task-understanding errors are only 0.9%.
- A single fired error code correlates with a 48% drop in overall score, and more scaffolding (Lite tier) can actually hurt performance -- GPT-5.4 drops 16.3% relative.
- The benchmark and execution harness are open-source. Live leaderboard and GitHub repo are publicly available.
A new benchmark from UC Santa Cruz and NVIDIA is asking a question that matters more than most: can today's AI agents actually do medical research, end to end, without a human holding their hand? The answer, according to AutoMedBench, is a qualified no -- and the reasons why are more interesting than you might expect.
Autonomous agents are increasingly expected to support end-to-end medical AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. AutoMedBench was built to fix that blind spot.
The gap no one was measuring
Most medical AI benchmarks today test whether a model can answer a multiple-choice question or classify an image. That is useful, but it is not research. Real research means picking a method, setting up an environment, running a pilot test, catching bugs, running full inference, and submitting results in the right format. AutoMedBench organizes agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. Every agent, every task, every run goes through the same pipeline.
AutoMedBench is a joint effort between University of California, Santa Cruz and NVIDIA, benchmarking autonomous agentic models across the medical AI research evaluation pipeline. The benchmark is openly available, with a live leaderboard and a public GitHub repository containing the full execution harness and sandbox.
What agents are actually being tested on
The benchmark evaluates 24 tasks across five medical research tracks, with each task assessed under two difficulty tiers, Lite and Standard, yielding 48 task-tier settings in total. The five tracks cover the major artifact types in medical imaging research:
- Segmentation -- producing voxel masks (e.g., kidney tumors, fetal brain tissue, pancreas), scored with macro Dice
- Image Enhancement -- low-dose CT denoising and MRI super-resolution, scored with SSIM
- Visual Question Answering -- radiology, pathology, and multi-frame medical VQA, scored by exact-match accuracy