BreastGPT Tackles Full Cancer Care With One Model 33x Faster
BreastGPT is the first unified multimodal LLM to handle the full breast cancer clinical pipeline, from screening to treatment planning, across five imaging modalities in a single model.
- BreastGPT is the first unified multimodal LLM covering the full breast cancer clinical workflow, from screening through diagnosis to treatment planning, in a single model.
- The team built BreastStage, a 1.86M instruction-following pair dataset from 17 sub-datasets across 5 imaging modalities (mammography, ultrasound, CT, MRI, pathology) and 136 task templates.
- A dual-branch visual encoder with concept-preserving token compression bridges the scale gap between standard radiology and gigapixel whole-slide pathology images, delivering a 33x inference speedup.
- On the held-out BreastStage-Bench benchmark, BreastGPT achieves 75.66% closed-ended accuracy and 89.92% open-ended score, outperforming both general-purpose and medical-specific MLLMs.
- The model is built on Qwen3-VL and uses stage-aware system prompts as lightweight task routers, avoiding task-specific heads while operating across all clinical stages.
- All data, code, and model checkpoints are publicly released; the paper is on arXiv.
Breast cancer clinical care is not a single task. It is a chain of decisions, each demanding a different kind of image and a different kind of reasoning. A radiologist reading a mammogram during screening is doing something fundamentally different from a pathologist examining a gigapixel whole-slide image (WSI) to plan surgery. Yet until now, AI models for breast cancer have been built to handle only one link in that chain at a time. BreastGPT changes that.
BreastGPT is a unified multimodal LLM that handles mammography, ultrasound, CT, MRI, and gigapixel pathology within a single instruction-following framework, enabling consistent reasoning across the full breast cancer clinical routine. The result is the first model capable of operating across the entire care continuum, from early detection to treatment planning, without swapping out specialized tools at each stage.
Why one model for everything is harder than it sounds
Clinical management of breast cancer requires multimodal reasoning across a workflow that spans screening, diagnosis, and treatment planning, where each stage involves distinct imaging modalities, task objectives, and reasoning patterns. The imaging alone spans an enormous range of scales and data types. A standard mammogram is a 2D X-ray image. An MRI scan is a 3D volumetric series. A whole-slide pathology image (WSI) is a gigapixel-resolution scan that can be tens of thousands of pixels wide.
Constrained by data scarcity and model versatility, existing medical MLLMs are typically evaluated on isolated modalities or narrow task families, limiting their ability to support workflow-level clinical reasoning. The scale mismatch between standard radiology images and WSIs is a particularly acute problem: you cannot feed a gigapixel image through a standard vision transformer without either running out of memory or throwing away most of the detail.
There is also a standardization challenge. Radiologists use a scoring system called BI-RADS (Breast Imaging Reporting and Data System) to classify findings, where scores range from 1 (negative) to 6 (pathology-proven malignancy). BI-RADS 4 has three sub-categories (4a, 4b, 4c) referring to suspicious findings with 2-10%, 10-50%, and 50-95% likelihood of malignancy, respectively, while BI-RADS 5 refers to suspicious findings with more than 95% likelihood of malignancy. Getting a model to reason correctly about these distinctions across five different imaging modalities is a non-trivial alignment problem.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
