Skip to main content
Back

Max Planck's SeSAM Matches 94% of Full Supervision With 2% Annotation Cost

A new framework adapts SAM for weakly supervised semantic segmentation, hitting 94% of full supervision quality using just 2% of annotation budget.

Pro

Paper, Full Access
  • SeSAM adapts SAM for weakly supervised semantic segmentation using points, scribbles, or coarse masks. Paper
  • Scribble supervision reaches 94% of fine-label performance with only 2% of the annotation budget on Cityscapes.
  • Three-step recipe: instance separation via connected components, skeleton-based point sampling, weak-label-aware mask selection.
  • Trained in a FixMatch-style semi-supervised loop combining weak labels, SAM masks, and teacher pseudo-labels.
  • Original SAM ViT-B beats SAM 2 and SAM 3 on this refinement task, hitting 74.7 F1 vs 74.2 and 73.9.
  • Half the training time and a third the GPU memory of prior state of the art AGMM, with higher mIoU.

Labeling images at the pixel level is one of the most tedious jobs in computer vision. A single Cityscapes image takes about 90 minutes to annotate finely, which is why the field keeps hunting for shortcuts. A new paper from the Max Planck Institute for Informatics, Do Instance Priors Help Weakly Supervised Semantic Segmentation?, proposes SeSAM, a framework that bends the Segment Anything Model into a workable semantic segmentation trainer using only points, scribbles, or coarse masks.

The headline number: scribble supervision matches 94% of fully supervised performance while using only 2% of the annotation budget. That shifts the price-performance curve meaningfully for anyone building a segmentation dataset from scratch.

SeSAM annotation types and Cityscapes mIoU vs annotation cost plot

Why SAM does not just solve this out of the box

Segment Anything (SAM) is a foundation model for promptable segmentation, but it segments instances, not classes. Semantic segmentation needs every pixel labeled with one of N classes, and a single class like "person" can span many disconnected blobs across the image. Sampling class points naively from class masks introduces ambiguity between different instances, causing SAM to smear a single mask across all sampled points and produce noisy boundaries.

Two other landmines lurk in the pipeline. SAM produces three candidate masks per prompt at different levels of granularity (whole object, part, subpart), each with a confidence score, and the highest-scoring one is often wrong for semantic segmentation. Prompt placement also matters a lot: sampling based on network confidence, as prior work does, tends to cover only part of an object.