Skip to main content
Back

DiTTA Turns Meta's SAM2 Into a One-Time Teacher, Running 10x Faster

A new test-time adaptation method turns any image segmenter into a video segmenter by distilling SAM2's temporal knowledge from just the first frames.

Pro

Paper, Full Access
  • DiTTA converts any image segmentation model into a video model at test time, no video labels required.
  • It distills SAM2's temporal knowledge in a single pass over the first ~10% of frames.
  • Adds a lightweight temporal fusion module plus contrastive alignment with a momentum model.
  • On VSPW: 51.1 mIoU (+2.1) and mVC16 92.2 (+7.9) vs SegFormer baseline, at 13.45 FPS.
  • Roughly 10x faster than zero-shot SAM2 refinement (1.41 FPS) with comparable temporal consistency.
  • Accepted at CVPR 2026; no public code released yet.

Getting per-pixel labels for every frame of a video is one of the most expensive annotation jobs in computer vision. A new method called DiTTA (Distillation-assisted Test-Time Adaptation), accepted to CVPR 2026, sidesteps the problem entirely by taking any pre-trained image segmenter, watching the opening seconds of a test video, and quietly upgrading itself into a temporally consistent video segmenter with no video labels required.

The trick is to borrow temporal reasoning from SAM2 (Meta's video-capable Segment Anything model) through knowledge distillation, and to do it only once at the start of each clip rather than at every frame during inference.

Why frame-by-frame segmentation keeps failing

Two standard approaches dominate video segmentation today, and both carry painful trade-offs. Applying an image model frame by frame circumvents expensive annotated video datasets, but each frame is processed in isolation and temporal continuity is discarded. Moving objects follow coherent trajectories that provide strong cues under occlusion, and neglecting them produces flickering masks, labels that swap between adjacent frames, and objects that vanish the moment they are partially hidden.

Fully supervised video semantic segmentation is the other option, but datasets like VSPW demand extensive pixel-level labeling and substantial human effort. A recent alternative bolts SAM2 onto an image segmenter as a post-processor. In this zero-shot refinement setup, object-level visual prompts extracted from the initial output are propagated temporally using SAM2. It works, but running a foundation model on every frame is slow and cannot fix the underlying model's blind spots.