Microsoft's PRISM2 Detects Any Cancer With a Simple Yes or No
PRISM2 is a 4.4B-parameter pathology foundation model that matches specialized cancer detectors using plain question-answering, with weights freely available for research.
- PRISM2 is a 4.4B-parameter multimodal pathology foundation model from Microsoft Research and Paige, trained on 2.3M whole-slide images paired with 685K clinical reports.
- It replaces task-specific cancer detection models with a single model that answers plain yes/no questions, reaching 0.880 balanced accuracy on pan-cancer detection vs. 0.728 for prior CLIP-style approaches.
- A two-stage training pipeline uses contrastive alignment first, then unlocks Phi-3 Mini (3.8B) for clinical dialogue fine-tuning on 14M GPT-4o-generated question-answer pairs.
- Two embedding types are exposed: a "base" embedding (best for biomarkers) and a "diagnostic" embedding from the LLM hidden state (best for cancer detection, 0.965 AUC).
- Weights are free for non-commercial academic research on Hugging Face (CC-BY-NC-ND 4.0); a survival-prediction variant is at paige-ai/Prism2-survival.
- Key limitations: trained only on MSKCC data, no IHC/molecular support, and biomarker prediction does not significantly outperform prior models.
Most AI systems built for pathology are single-purpose: one model detects prostate cancer, another flags breast lymph node metastasis, and a third handles something else entirely. Every new clinical question means rebuilding from scratch. PRISM2, a collaboration between Microsoft Research and Paige (now part of Tempus), trains one model on the language of pathology itself and lets it answer any question you ask.
One model, many tasks
Pathology is both a visual and a language-driven discipline. Radiologists write reports; pathologists write reports. So instead of training on images alone, the PRISM2 team fused both modalities from the start, pairing over 2.3 million whole-slide images with their corresponding clinical reports. A whole-slide image (WSI) is a gigapixel-scale digital scan of a tissue sample, the kind a pathologist examines under a microscope. Processing them at scale is a serious engineering challenge, and grounding them in diagnostic text is what separates PRISM2 from earlier vision-only foundation models.
How it was built
PRISM2 builds on Virchow2, trained on more than 2.3 million H&E-stained whole-slide images spanning hundreds of thousands of patient cases. H&E staining (hematoxylin and eosin) is the standard tissue preparation for microscopy, coloring cell nuclei blue-purple and cytoplasm pink. Each slide is paired with its clinical report, grounding the model in real-world diagnostic language.
Training happened in two stages:
- Stage 1, vision-language alignment: A perceiver-based slide encoder (620M parameters) trains with a contrastive objective to align whole-slide embeddings with clinical report text encoded by BioGPT. The Phi-3 Mini LLM stays frozen, but the autoregressive loss runs to pre-align the adapter.