Synthetic Hospital tests AI on whole-chart reasoning

Clinical AI benchmarks often measure isolated medical recall through licensing exams and knowledge tests. Those scores reveal little about whether a model can connect diagnoses, findings, and treatments across months of a patient’s chart. A new Carnegie Mellon paper targets that longitudinal task with artificial records; in a small blinded review, physicians classified its synthetic and real charts with 53% accuracy, close to chance.

Synthetic Hospital contains 1,268 patients and 5,602 encounters derived from public medical education material. Every diagnosis, finding, and temporal relationship is grounded in ICD-10-CM, SNOMED CT, or LOINC, and the dataset contains no protected health information. The code and benchmark materials are available in a GitHub repository.

Real charts hide the answer key

Real electronic health records create two obstacles for benchmark designers. Privacy rules, institutional review board requirements, and data-use agreements restrict access and redistribution. De-identified datasets such as MIMIC-IV, a critical-care EHR corpus, still require controlled access.

Clinical documentation also captures what clinicians recorded at the time, leaving diagnoses, findings, and causal relationships unstated or ambiguous. When a model produces the wrong answer, evaluators may struggle to separate model failure from missing source information or an incomplete reference label.

Synthetic Hospital starts with questions and reference material modeled on the United States Medical Licensing Examination. A structured graph records the complete clinical state, and a renderer converts those facts into realistic notes and encounters. The simulated hospital serves records through FHIR R4, a standard interface for exchanging healthcare data, with role-based access, Epic-style workflows, and tools that agents can call programmatically.