Skip to main content
Back

Toronto Radiologists Reveal Why AI Never Makes It Into the Reading Room

A survey of 46 radiologists reveals a wide gap between the explainability tools researchers build and what clinicians actually need to trust a model.

Pro

Paper, Full Access
  • Survey of 46 radiologists finds current XAI tools miss what clinicians actually need to trust ML models. Read the paper.
  • Top explainability requirement is focusing on clinically relevant features (86% of senior radiologists), not fancy explanation methods.
  • 86% of senior radiologists want structured scoring like BI-RADS in model outputs; zero residents selected this option.
  • Radiologists want ML for workflow management and routine tasks, not for diagnosing rare or difficult cases.
  • Counterfactual explanations, popular in XAI research, were the least wanted format at just 33%.
  • Authors propose a four-stage guideline set favoring self-explainable multimodal models over post-hoc heatmap tools.

Machine learning models keep matching radiologist performance in papers, then failing to make it into the reading room. A new study from the University of Toronto and Hospital for Sick Children asks the actual end users why, and the answers challenge assumptions baked into current explainable AI research.

The team ran a 30-question structured survey with 46 clinicians across multiple hospitals in one metropolitan area, split between 26 staff radiologists and 20 residents, then translated the findings into concrete design guidelines for building clinically useful models. You can read the full paper on arXiv.

The trust gap nobody is closing

Most explainability methods are designed by ML researchers for ML researchers. Popular post-hoc tools like Grad-CAM and LIME have mostly been validated with automated metrics that may not align with clinical expectations or the explainability requirements of medical processes. The Toronto group set out to fix that by running a survey covering multiple modalities (CT, MRI, X-ray) and multiple clinical specialties rather than a single use case.

Participants were roughly split between the most experienced group (30% with more than 10 years) and residents (43%), which let the authors compare how expectations shift with seniority. Notably, 52% of respondents had no significant ML experience, so the survey captures the perspective of the users the field needs to convince, rather than AI enthusiasts.

Diagram of five explainability methods for medical imaging: textual, attention map, semantic features, prototype, and counterfactual

What radiologists actually want from an explanation

The most striking finding is that clinicians want several forms of explanation at once, and the priority list differs from what most papers optimize for: