Microsoft's PRISM2 Detects Any Cancer With a Simple Yes or No

Most AI systems built for pathology are single-purpose: one model detects prostate cancer, another flags breast lymph node metastasis, and a third handles something else entirely. Every new clinical question means rebuilding from scratch. PRISM2, a collaboration between Microsoft Research and Paige (now part of Tempus), trains one model on the language of pathology itself and lets it answer any question you ask.

One model, many tasks

Pathology is both a visual and a language-driven discipline. Radiologists write reports; pathologists write reports. So instead of training on images alone, the PRISM2 team fused both modalities from the start, pairing over 2.3 million whole-slide images with their corresponding clinical reports. A whole-slide image (WSI) is a gigapixel-scale digital scan of a tissue sample, the kind a pathologist examines under a microscope. Processing them at scale is a serious engineering challenge, and grounding them in diagnostic text is what separates PRISM2 from earlier vision-only foundation models.

How it was built

PRISM2 builds on Virchow2, trained on more than 2.3 million H&E-stained whole-slide images spanning hundreds of thousands of patient cases. H&E staining (hematoxylin and eosin) is the standard tissue preparation for microscopy, coloring cell nuclei blue-purple and cytoplasm pink. Each slide is paired with its clinical report, grounding the model in real-world diagnostic language.

Training happened in two stages:

1. Stage 1, vision-language alignment: A perceiver-based slide encoder (620M parameters) trains with a contrastive objective to align whole-slide embeddings with clinical report text encoded by BioGPT. The Phi-3 Mini LLM stays frozen, but the autoregressive loss runs to pre-align the adapter.
2. Stage 2, clinical dialogue fine-tuning: The slide encoder is frozen and Phi-3 Mini (3.8B parameters) is unfrozen. The model trains for 16 epochs across four chat templates: report generation, yes/no QA, open-ended QA, and image-text matching.

To generate training signal without new annotations, the team used GPT-4o to convert clinical reports into 14 million question-answer pairs, turning existing medical records into structured supervision. The total model runs to 4.4B parameters: a 620M perceiver for slides, a 3.8B Phi-3 Mini decoder, and a 29M MLP adapter bridging the two. Training ran on 56 A100 40GB GPUs.

The yes/no approach that changes classification

The most practically significant result in the paper is a new method for zero-shot classification. Earlier multimodal pathology models used CLIP-style contrastive scoring: embed the image, embed a text prompt like "malignant tumor," and rank similarity. This approach is brittle. You need to hand-craft both positive and negative prompts, and performance swings with word choice.

PRISM2 replaces that with simple yes/no questions. Feed the model a slide and ask: "Is invasive carcinoma present?" The model returns a probability over "Yes" and "No" tokens, with no prompt engineering and no exhaustive list of negative classes.

On pan-cancer detection across 16 tissue origins, the numbers are stark:

• TITAN contrastive zero-shot: 0.728 balanced accuracy
• PRISM2 contrastive zero-shot: 0.719 balanced accuracy
• PRISM2 yes/no dialogue: 0.874 balanced accuracy
• PRISM2 dialogue with generated report as context: 0.880 balanced accuracy

The jump from contrastive to dialogue-based classification is largest on rare cancer types and low-sample-size scenarios, exactly the cases where task-specific models are hardest to train.

Two embeddings for two use cases

PRISM2 exposes two types of slide-level embeddings, and the distinction matters for anyone building on top of it:

• Base embedding (2560-dim): Produced by the perceiver's attention pooling. Best for biomarker prediction and out-of-distribution generalization, since it captures lower-level visual features unconstrained by diagnostic language.
• Diagnostic embedding (3072-dim): Extracted from Phi-3's hidden state after processing the slide. Better for cancer detection and subtyping, where the LLM's reasoning sharpens the representation toward clinically meaningful distinctions.

On pan-cancer detection, the diagnostic embedding reaches 0.965 AUC versus 0.952 for the base embedding and 0.931 for TITAN. On biomarker prediction, the base embedding wins, because the LLM was never trained to discuss molecular markers and its hidden state carries less signal there.

Running a yes/no classification query with the Hugging Face model looks like this:

import torch
from transformers import AutoModel, AutoProcessor
model = AutoModel.from_pretrained(
    "paige-ai/Prism2", trust_remote_code=True, torch_dtype="auto"
).cuda().eval()
processor = AutoProcessor.from_pretrained("paige-ai/Prism2", trust_remote_code=True)
# slides: list of (N_tiles, 1280) Virchow2 tile embeddings
batch = processor(tile_embeddings=slides).to("cuda")
with torch.autocast("cuda", torch.bfloat16):
    scores = model.yes_no_score(
        tile_embeddings=batch["tile_embeddings"],
        attention_mask=batch["attention_mask"],
        question="Is invasive carcinoma present?",
    )
# scores: P(Yes) per slide in batch

Known limits

PRISM2 has clear gaps worth knowing before building on it. The model does not handle immunohistochemistry (IHC) or molecular assay results, both of which were excluded from training. Biomarker prediction performance is competitive but does not clearly beat prior models. The training data came entirely from Memorial Sloan Kettering Cancer Center, leaving open questions about generalization to different staining protocols, scanner types, and patient demographics. The authors note that fairness evaluation could not be completed due to metadata limitations.

Contrastive zero-shot performance also remains sensitive to prompt wording in ways the yes/no approach avoids, a failure mode the paper documents in detail for breast tissue pan-cancer detection.

Where the field is heading

Pathology AI has been moving from tile-level models, which analyze small image patches, toward whole-slide models that understand full tissue context. PRISM2 extends that further by adding a conversational interface on top. The same model weights can generate a diagnostic report, answer a targeted question, or produce an embedding for a downstream classifier, covering diagnostics, biomarker prediction, and multi-modal patient outcome modeling without retraining.

The model weights are available on Hugging Face under a CC-BY-NC-ND 4.0 license for non-commercial academic research, with a separate survival-prediction variant at paige-ai/Prism2-survival. Commercial use requires separate approval from the authors. The paper is published in Nature Medicine.