RWTH Aachen Finds BioViL-T Re-Links 'Anonymous' X-Rays to Patient Reports 50x Above Chance
Clinical vision-language models can re-link de-identified chest X-rays to their original radiology reports using cosine similarity alone — and a targeted privacy fix cuts the risk by 62% without hurting diagnostic utility.
Takeaways
- Core finding: Clinical VLMs can re-link de-identified chest X-rays to their original radiology reports via cosine similarity, with risk rising as models become more specialized.
- Scale of risk: The strongest model (BioViL-T) retrieved the correct report at 50x chance in a pool of 10,000 candidates on MIMIC-CXR.
- Not just disease matching: The signal persisted under pathology-matched hard negatives, meaning models encode patient-specific features beyond broad diagnostic categories.
- The fix: Freezing both encoders and applying differentially private optimization (ε=0.34) only to the projection heads cut Recall@1 by 61.8% at N=10,000.
- Utility preserved: Macro AUROC for 14-label pathology classification shifted only from 79.63% to 79.43% after the privacy intervention.
- No code released yet; paper available at arxiv.org/abs/2606.02276 with methodology reproducible on public MIMIC-CXR and CheXpert Plus datasets.
There is a quiet assumption baked into how hospitals share medical imaging data: if you strip the name off a chest X-ray and keep the radiology report locked away in a separate system, the patient is protected. A new paper from researchers at RWTH Aachen University and University Hospital Düsseldorf challenges that assumption head-on. Their finding is uncomfortable , the more clinically specialized a vision-language model (VLM) is, the better it becomes at re-linking a de-identified image back to its original narrative report, using nothing more than a cosine similarity search in the model's shared embedding space.

The problem hiding in plain sight
Modern clinical VLMs , think BioViL-T, BiomedCLIP, and PubMedCLIP , are trained on paired chest X-rays and radiology reports. The training objective is contrastive: push the image and its matching report close together in a shared vector space, and push non-matching pairs apart. This is exactly what makes them useful for zero-shot classification, report retrieval, and cross-modal search. But it is also what creates the privacy risk.
HIPAA prescribes two options for de-identification of medical data: removal of 18 safe harbor identifiers, or certification by an expert that the risk of re-identification is low. In practice, many institutions share imaging data without the accompanying reports, treating physical or access-controlled separation as a sufficient privacy barrier. The paper formalizes the attack scenario: given a de-identified X-ray and a database of reports, can a VLM retrieve the correct report for that image? The answer, it turns out, is yes , and the risk scales with how specialized the model is.
How bad is it, exactly?
The researchers evaluated four VLMs of increasing clinical specialization , generic CLIP, PubMedCLIP, BiomedCLIP, and BioViL-T , on over 406,000 paired examples drawn from 126,804 patients. They used two public datasets: MIMIC-CXR, a large-scale de-identified chest radiograph dataset from Beth Israel Deaconess Medical Center that includes frontal and lateral X-rays along with corresponding free-text radiology reports , and the external CheXpert Plus dataset as a cross-institution generalization test.