Tencent's SAM3-I Beats an 8.8B Agent Stack With Just 1.1B Parameters
A new adapter mechanism lets SAM3 directly parse complex referring and reasoning instructions in one pass, without an external multimodal agent.
ProTakeaways
- SAM3-I extends Meta's SAM3 to follow complex natural-language instructions in a single pass, without an external MLLM agent.
- A cascaded S-Adapter and C-Adapter inject instruction reasoning into SAM3's frozen text encoder and detector, adding ~0.3B trainable params.
- Four alignment losses regularize the two branches at both embedding and mask level to prevent semantic drift.
- Beats the 8.8B SAM3 Agent baseline by +31 gIoU on simple and +22 gIoU on complex instructions using 1.1B params.
- New HMPL-Instruct dataset covers 44K images, 849K instructions, plus object- and part-level masks.
- Code, dataset, and annotation tooling released open-source; concept-level performance is preserved by construction.
Meta's Segment Anything Model 3 was a big leap for open-vocabulary segmentation, but it has a curious blind spot: it only really understands short noun phrases. Ask it for "soccer player" and it will happily mask every player on the field. Ask for "the soccer player on the far right, wearing a blue jersey, sliding to tackle the ball," and the model has to phone a friend, typically a multimodal LLM, to boil that instruction back down into a noun phrase it can actually consume.
A team from the University of Alberta, Tencent WeChat, Yale, and several other labs has now released SAM3-I, an instruction-following extension that teaches SAM3 to parse those richer sentences directly, in a single forward pass, while keeping its original concept-segmentation abilities intact.
The noun-phrase ceiling
SAM3 supports Promptable Concept Segmentation, so you can find and segment every instance of a concept using simple text descriptions like "yellow school bus." Real user instructions rarely look like that, though. They combine attributes, spatial relations, actions, states, and implicit reasoning. The paper frames three levels of difficulty:
- Concept: short noun phrases like "soccer player," what SAM3 already handles.
- Simple: still names the target but adds conditions, like "the player in the blue jersey on the far right."
- Complex: drops the noun entirely and forces reasoning, like "the one sliding on the grass to tackle the ball."

The workaround so far has been an agent pipeline: an external MLLM like Qwen3-VL rewrites the instruction into a noun phrase, SAM3 segments, and the MLLM re-checks the masks over multiple filtering rounds. It works, but it separates language reasoning from visual grounding, balloons the parameter count, and tends to collapse detailed queries into oversimplified concepts.