Skip to main content
Back

Michigan Finds BiomedCLIP's Pooling Choice Breaks 3D Medical AI

University of Michigan researchers publish the first vision-language transfer learning baselines for CT enterography, showing that slice pooling strategy dictates task performance.

  • First vision-language transfer learning study on abdominal CT enterography, using 1,074 exams.
  • Mean pooling wins classification (59.2% accuracy); attention pooling wins retrieval (0.235 MRR) across every LoRA setting.
  • Multi-window RGB encoding beats adding coronal and sagittal views, which actually hurt classification performance.
  • Fine-tuned MedGemma alone hits severity accuracy at chance level; RAG lifts it 7 to 14 points above chance.
  • Three-teacher pseudolabel ensemble (rules + BioMistral + Qwen2.5) enables training without expert annotations.
  • Code released at github.com/Minoch/RadIBD as baseline for volumetric medical vision-language work.

A team from the University of Michigan has published what they describe as the first study of vision-language transfer learning on abdominal CT enterography, and the headline finding is not a leaderboard number. It is a geometry problem: the way you pool slice embeddings inside a 3D volume decides which downstream task your model will be good at, and the same pooling choice cannot win at both classification and retrieval.

CT enterography is a workhorse imaging modality for inflammatory bowel disease, but unlike chest scans, no large public benchmarks exist for this modality, and no prior work has applied vision-language contrastive learning to abdominal CT with paired radiology reports. That gap forces basic representational questions to be answered from scratch, which is exactly what this paper does across 1,074 examinations from Michigan Medicine.

The pooling trade-off nobody was measuring

The authors adapt BiomedCLIP using a 2.5D slice-based approach, then systematically swap the aggregator that turns per-slice embeddings into a single volume embedding. Three options were tested: mean pooling, attention pooling with a learnable query, and a lightweight transformer encoder with a CLS token.

The result held across every LoRA configuration they tried:

  • Mean pooling of slice embeddings gives better categorical disease assessment (59.2% three-class accuracy), but weaker retrieval.
  • Attention pooling gives better cross-modal retrieval (0.235 text-to-image MRR), but weaker classification.