TCSA-UDA Uses Text to Bridge the CT-MRI Gap, Hitting 82.4 Dice
A new domain adaptation framework uses medical text descriptions like 'left atrium' to bridge the gap between CT and MRI segmentation models without target labels.
ProTakeaways
- TCSA-UDA aligns CT and MRI segmentation features using text embeddings of anatomical class names as domain-invariant anchors.
- Novel Vision-Language Covariance Cosine Loss matches inter-class relationships between pixel features and text embeddings, not features directly.
- Achieves 82.4 Dice on cardiac MRI to CT, closing 88.1% of the gap to supervised performance from a 23.3 baseline.
- Modality-aware prompts like 'A cardiac CT/MRI imaging of a [CLS]' outperform generic CLIP prompts by 3-4 Dice points.
- Consistent gains on MMWHS cardiac, CHAOS abdominal, and BraTS brain tumor benchmarks across bidirectional adaptation.
- Code available at github.com/lalitmaurya47/TCSA_UDA with DeepLabV2 backbone and CLIP/BioBERT text encoders.
Training a segmentation model on CT scans and then deploying it on MRI (or vice versa) usually ends badly. Pixel intensities look nothing alike, scanners disagree, and hospitals rarely have the labeled data to fine-tune for every new modality. A new paper from researchers at the University of Portsmouth and Aberystwyth University proposes a workaround that leans on the words themselves. Their framework, TCSA-UDA, uses text embeddings of anatomical class names to anchor visual features across modalities, and it beats a broad slate of existing domain adaptation baselines on cardiac, abdominal, and brain tumor benchmarks.
The domain shift problem, in one sentence
Fully-supervised deep learning methods perform well in-domain but are sensitive to domain shifts, which are common in real-world medical data, and annotating fresh examples for each new domain is labor-intensive and often impractical in clinical workflows. Unsupervised domain adaptation (UDA) tries to transfer knowledge from a labeled source modality to an unlabeled target modality, though most methods work purely in visual space, either translating image appearance (CycleGAN-style) or aligning feature distributions with adversarial losses.
The authors' core observation is straightforward: unlike visual features, textual class semantics like "left atrium" or "tumor" are inherently domain-invariant, because they describe anatomical entities that remain consistent across modalities. A left ventricle is a left ventricle whether you image it with protons or X-rays. The text embedding, then, can serve as a shared anchor that pulls CT and MRI features into the same semantic neighborhood.