Georgia Tech's BALD-SAM Beats Human Annotators on 14 of 16 Benchmarks

Meta’s Segment Anything Model (SAM) converts point, box, or mask prompts into segmentation masks. Point-guided annotation often becomes iterative: an annotator clicks an object, inspects the mask, and adds corrective clicks until the boundary is usable. Researchers at Georgia Tech’s OLIVES lab developed BALD-SAM to recommend the pixel expected to provide the most information when the annotator labels it as foreground or background.

The paper treats each click as an active-learning query within one image. On the PointPrompt benchmark, BALD-SAM outperformed recorded human prompt sequences on most tested datasets and ranked first or second on 14 of 16 benchmarks. The results suggest that uncertainty-guided placement can reduce the clicks required to improve a mask when SAM already represents the target domain well.

Every click becomes an uncertainty query

Interactive segmentation conditions each new prediction on the image and all previous prompts. BALD-SAM scores every candidate pixel after each interaction, selects the location with the highest expected information gain, and asks for its foreground or background label. The new labeled point joins the prompt set before the system recomputes every score.

Predictive entropy alone gives high scores to two sources of uncertainty. Epistemic uncertainty reflects limited model knowledge, which additional evidence can reduce. Aleatoric uncertainty comes from ambiguous image content, such as a blurred boundary or image noise, and can remain high after another click.

Bayesian Active Learning by Disagreement, or BALD, targets epistemic uncertainty through the mutual information between predictions and model parameters. For each pixel, the score is the entropy of the average prediction across sampled models minus the average entropy of their individual predictions. A high score indicates that the sampled models are individually confident yet disagree with one another.