Georgia Tech's fMRI-LM Teaches an LLM to Read Brain Scans Like Text
A three-stage framework translates brain scans into tokens a language model can read, hitting 94.9% on sex classification and adapting from 4 examples.
Takeaways
- fMRI-LM aligns brain scans with LLMs via a three-stage tokenize, align, instruction-tune pipeline (paper)
- Pretrained on 50,000+ scans across UK Biobank, ABCD, and five clinical datasets spanning ages 5 to 100
- Solves the missing-captions problem by synthesizing text from FC, ICA, graph, and gradient features
- Hits 94.89% on UKB sex, 76.56% on ABIDE2 autism, beating Brain-JEPA and BrainLM baselines
- Four-shot adaptation matches full-dataset training on several tasks; LoRA works as well as full fine-tuning
- Code and checkpoints on GitHub; cannot yet explain which brain regions drive predictions
Brain imaging has a language problem. Deep learning models can predict age or diagnose disease from functional MRI scans, but each task typically demands its own labeled dataset and custom architecture. Multimodal LLMs have swallowed images, audio, and video into a single reasoning framework, yet brain signals remained locked out. A new foundation model called fMRI-LM tries to close that gap by teaching a language model to read brain activity as if it were text.
The system, from researchers at Georgia Tech's TReNDS Center and collaborators, was accepted to CVPR 2026 and pretrained on more than 50,000 scans. It handles disease diagnosis, phenotype prediction, and free-form description generation through a single instruction-tuned interface, adapting to new tasks from as few as four labeled examples.
Why brain scans resist the LLM treatment
Existing fMRI foundation models like BrainLM and Brain-JEPA pretrain on large neuroimaging corpora and transfer well to downstream tasks, yet they remain confined to neural-only objectives such as masked prediction and contrastive learning, require task-specific tuning, and lack grounding in language. Without a linguistic scaffold, these models can produce useful embeddings but cannot describe what a scan shows, answer arbitrary questions about it, or generalize across studies that use different labels.
The obvious fix, borrowed from vision-language models, would be to train on paired fMRI-caption data. No such corpus exists. Radiologists do not routinely caption resting-state scans the way photographers caption images, and the raw BOLD signal, a 4D tensor of blood-oxygen fluctuations across the brain, has no natural textual counterpart.