Iran University Fixes MedSAM's Fatal Flaw With a 1.6M-Parameter Click-to-Box Trick

Getting an AI to accurately outline a tumor or organ in a medical scan is harder than it looks. The images are low-contrast, the structures are irregular, and the models are notoriously sensitive to how you tell them where to look. A new paper from Iran University of Science and Technology tackles one of the most persistent pain points in this space: the fragility of point-based prompting in MedSAM, the leading foundation model for medical image segmentation.

Why a single click isn't enough

MedSAM is a fine-tuned version of Meta's Segment Anything Model (SAM), trained on over 200,000 medical masks spanning 11 imaging modalities. While MedSAM improves over the original SAM by training on diverse medical imaging data, the heterogeneity across imaging modalities, anatomical regions, and acquisition protocols means that further domain-specific fine-tuning is often necessary to achieve optimal performance on particular clinical tasks.

The deeper problem is prompt design. SAM-based models are highly sensitive to the quality of input prompts, with this issue becoming particularly severe in medical imaging where regions of interest are often small, irregular, or poorly contrasted. Point prompts , where a user clicks somewhere inside the target region , are the most natural interaction, but they give the model almost no spatial context. A single dot in the middle of a brain MRI slice could mean a tumor, a ventricle, or healthy tissue. The model has to guess the extent of what you want.

There's also a subtler technical trap. MedSAM was fine-tuned with bounding box prompts as the primary prompt type, so its prompt encoder and mask decoder have learned to expect spatial inputs in this form. When you give it a point instead, you're not just providing less information , you're giving it the wrong kind of information entirely. The ablation results are stark: when MedSAM is used without any retraining, relying solely on a point prompt results in very poor performance across all datasets, with Dice scores ranging from 1.64 to 10.53.

One click, one box

The proposed fix is elegant in its simplicity. The team introduces a Box Predictor module: a small MLP (multi-layer perceptron , a basic feedforward neural network) that takes a single user click and automatically estimates a bounding box around the target structure. That predicted box is then fed into MedSAM's prompt encoder, restoring the geometric context the model was trained to expect.