Lund University Finds Meta's SAM 3 Loses to SAM 2 on Eye Tracking
A head-to-head evaluation of Meta's newest Segment Anything Model on eye-tracking segmentation finds that the older SAM 2 wins on accuracy and speed.
ProTakeaways
- SAM 2 beats SAM 3 on eye segmentation across lab and in-the-wild datasets, with better accuracy and higher speed.
- SAM 3 with text prompts failed entirely for
irisandsclera, returning no segmentation at all. - Pupil signal noise (RMS-S2S) from SAM 3 visual prompts was 112% worse than SAM 2 on lab recordings.
- SAM 3 concept prompts had order-of-magnitude worse precision than either visual-prompt configuration.
- SAM 3 throughput drops below 1 fps on long videos even on workstation GPUs like the RTX Pro 6000.
- Authors released a SAM 3 fork supporting arbitrary-length videos at github.com/dcnieho/sam3.
When Meta shipped Segment Anything Model 3, the headline feature was Promptable Concept Segmentation: hand the model a short noun phrase like yellow school bus and get back masks for every matching object in an image or video. Meta claims SAM 3 doubles the accuracy of existing systems on both image and video PCS while improving prior SAM capabilities on visual segmentation. A new evaluation from Lund University tested that claim on a narrow but demanding domain, eye-tracking video, and the newer model lost.
The study compares SAM 2 and SAM 3 on segmenting the pupil, iris, sclera, and corneal reflection in eye images. Across controlled lab recordings and unconstrained in-the-wild footage, SAM 2 came out ahead on accuracy, speed, and reliability. The authors also released an adaptation of the SAM 3 codebase that handles arbitrarily long videos, available on GitHub.
Why eyes are a brutal stress test
Gaze estimation pipelines typically need to localize specific features (pupil edges, iris boundary, the glint reflected from the tracker's infrared illuminator) to feed into geometric or regression-based gaze models. Traditional deep learning approaches require large annotated datasets, which are expensive to build. That is why SAM 2 has been attractive: it can segment those features zero-shot with a handful of manually placed clicks.
SAM 3 promised to remove the last bit of manual effort. Instead of clicking on the pupil to define it, you could type pupil and let the model find every instance. That would also help with edge cases like blink recovery, where visual prompts from a previous frame become invalid.