KAIST's DC-TTA Splits SAM Into Specialists to Conquer Hard Segmentation
A new test-time adaptation method splits user clicks into coherent groups, adapts SAM separately per group, and merges the specialists.
ProTakeaways
- DC-TTA adapts SAM per-image by partitioning user clicks into coherent groups, each with its own adapted model.
- Per-group models are merged using task vectors, then briefly fine-tuned on the union of predicted masks.
- On TRASHCAN, failure rate at 90% IoU drops from 52.29% to 38.80% versus baseline SAM.
- Camouflaged object benchmarks CAMO and COD10K also show clear NoC and failure-rate improvements.
- The divide-and-conquer strategy alone, with no weight updates, already beats the baseline in ablations.
- Method also improves older interactive segmentation networks like RITM, FocalClick and MFP; code coming.
Interactive segmentation, the task of refining an object mask through a handful of user clicks, has become one of the flagship use cases for the Segment Anything Model (SAM). Anyone who has poked at a camouflaged snake, a coral reef, or a shadowy trash pile with SAM knows the drill: the first click gives you something reasonable, and every subsequent click seems to make things worse in a different place. A team from KAIST and Chung-Ang University has a new answer to that problem, and it treats the model itself as the thing that needs to be split up.
Their method, called DC-TTA, does test-time adaptation on SAM on a per-image basis, using the user's clicks as free supervision. Rather than stuffing every click into one model and hoping it sorts things out, DC-TTA groups clicks that agree with each other, spins up a separate adapted copy of SAM for each group, and then merges the copies back together into one predictor.
Why one model chokes on many clicks
SAM performs well in straightforward cases, but its effectiveness diminishes on complex structures such as camouflaged or multi-part objects. The usual fix is test-time adaptation (TTA): update the network's weights at inference time using whatever signal you can scrape from the user, without ever touching the original training data. For interactive segmentation, that signal is the clicks themselves. Each new click implies that the previous mask was wrong somewhere, so the fresh prediction can serve as a pseudo ground truth to nudge the model.