Skip to main content
Back

MedP-CLIP Lets Doctors Click on Scans to Get Smarter AI Diagnoses

A new medical vision-language model accepts points, boxes, and masks as prompts, letting clinicians steer CLIP toward specific lesions without losing global context.

Pro

Paper, Full Access
  • MedP-CLIP adds a feature-level prompt encoder to CLIP that accepts points, boxes, and masks for region-aware medical image understanding.
  • Pre-trained on 97.3M region-text pairs derived from 6.4M images in IMed-361M, spanning 15 imaging modalities.
  • Hits 92.45% on Br35H brain tumor classification with ViT-L/14, beating BiomedCLIP and BMC-CLIP.
  • As a SAM backbone, sets new SOTA on ISLES (73.14%) and TotalSegmentator MRI (81.55%) with under 1% of parameters fine-tuned.
  • Replacing LLaVA-Med's vision encoder pushes region-level VQA precision from 64.92% (MedPLIB) to 67.96%.
  • Still confuses morphologically similar adjacent organs in 2D slices without 3D anatomical priors.

Medical images rarely reward a global reading. A radiologist looking at a chest CT cares about a specific nodule, not the average appearance of the lung, and the standard CLIP recipe of pooling one embedding per image throws that spatial detail away. A new model called MedP-CLIP lets users point at what they want the encoder to look at, using clicks, boxes, or masks, while keeping the surrounding anatomy in view.

The paper reports state-of-the-art results across zero-shot classification, region-level retrieval, interactive segmentation, and medical visual question answering, all from a single visual backbone that plugs into existing CLIP-based pipelines without architectural surgery.

Why global CLIP falls apart in the clinic

Existing CLIP adaptations for medicine, such as BiomedCLIP, PMC-CLIP, and MedCLIP, sharpen global semantic alignment by pretraining on domain-specific image-text corpora, yet they remain global encoders with no native region awareness, which limits their utility in clinical scenarios that demand precise spatial reasoning. The obvious workarounds each break something important:

  • Cropping the region of interest discards crucial global context, which matters when a lesion is only meaningful relative to nearby organs.
  • Masking the background throws away potentially informative surrounding structures.
  • Alpha-CLIP fuses region information through an extra alpha channel, tying inference quality to the precision of the supplied mask, which is impractical when a clinician just wants to click.
  • Methods like R-LLaVA and MedPLIB bolt region awareness onto an LLM through task-specific pipelines that do not scale.