University of Catania's Brain-GraSP Fixes Where AI Hallucinates Brain-Decoded Images
Brain-GraSP predicts saliency maps directly from fMRI-CLIP embeddings using graph neural networks, then conditions a frozen Stable Diffusion model for sharper, more faithful scene reconstruction.
ProTakeaways
- Brain-GraSP adds a graph neural network that predicts saliency maps directly from fMRI-CLIP embeddings.
- Saliency mask plus extracted text cues condition a single frozen Stable Diffusion model via IP-Adapter.
- Reuses MindEye's precomputed embeddings, avoiding retraining and multi-stage diffusion sampling.
- GraphSAGE beat GCN and GAT as the saliency backbone thanks to inductive neighborhood aggregation.
- Tops MindEye, MindBridge, and BOI on PixCorr, SSIM, Inception, CLIP, and SwAV on NSD subjects.
- Subject-specific training and EDN-derived pseudo-ground-truth saliency remain the main limitations.
Reading pictures out of the brain has become one of the more surreal demos in machine learning. Feed an fMRI scan of someone looking at an image into a modern decoder, and you get back a passable regeneration of what they saw. The catch is that these reconstructions tend to hallucinate the layout: the right objects appear, but in the wrong places, at the wrong count, or floating in a scene that no longer resembles what the subject actually looked at.
A new framework called Brain-GraSP, from a team at the University of Catania, tries to fix that by adding an explicit spatial prior. Instead of hoping the diffusion model figures out where things go, it uses a graph neural network to predict a saliency map directly from brain activity, then uses that mask to steer image generation.
The layout problem in brain decoding
Recent fMRI decoders lean on two ingredients: CLIP-style embeddings to align brain signals with visual-semantic features, and diffusion models to turn those embeddings into images. The authors point out that fMRI-based visual brain decoding remains in its early stages and faces critical challenges, particularly in preserving object-level attributes such as saliency, count, and spatial arrangement in reconstructed scenes. A picture of two dogs on a beach might come back as a single dog in a field of grass, semantically close but structurally wrong.
Prior systems like MindEye, MindBridge, and Brain-Optimized Inference (BOI) attack this problem by scaling up embeddings or by running diffusion multiple times to pick the best sample. BOI refines MindEye reconstructions by iteratively sampling candidate images from a diffusion model and selecting those most consistent with measured fMRI activity, which works but is expensive. Brain-GraSP takes a different route by keeping the diffusion model frozen and single-pass, while injecting a better prior about where things belong.