StreamGaze Reveals Every AI Model Fails at Following Human Eye Attention
A new benchmark tests whether multimodal models can use eye-gaze to follow attention, infer intent, and predict actions in live egocentric video.
- StreamGaze is the first benchmark testing MLLMs on gaze-guided temporal reasoning in streaming egocentric video. Paper
- Contains 8,521 QA pairs across 10 tasks covering past, present, and proactive timestamps. Project
- Pipeline uses fixation extraction, FOV object detection via InternVL3.5-38B, and scanpath construction.
- GPT-4o, Claude Opus 4, InternVL-3.5, Qwen2.5-VL, and streaming models all trail human performance significantly.
- Proactive gaze tasks like Gaze-Triggered Alert expose the biggest weaknesses in current MLLMs.
- Follow-up GazeQwen achieves 63.9% by modulating the LLM internally rather than via prompts. Details
StreamGaze, introduced as a NeurIPS 2026 benchmark, tests whether multimodal large language models can use recorded eye gaze while processing egocentric video as a live stream. Across 8,521 question-answer pairs and 10 tasks, every evaluated model trails the human reference, with the largest gaps appearing on tasks that require anticipation.
Streaming evaluation requires a model to process frames in sequence and answer before future frames arrive. Gaze adds a continuous signal about which object or region holds the wearer’s attention. For developers building augmented-reality assistants, wearable agents, or monitoring systems, that signal could help resolve ambiguous objects, infer intent, and trigger timely responses.
Why gaze changes streaming video
Earlier benchmarks such as StreamingBench showed that frontier models lag behind humans on online video question answering. Those evaluations measured temporal reasoning without testing whether models could interpret a viewer’s eye movements.
Head-mounted systems such as Meta’s Aria glasses and Apple Vision Pro use gaze for sensing or interaction. A gaze-aware assistant must associate a sequence of coordinates with objects and actions over time, then use that association while generating an answer. StreamGaze evaluates that full chain.
From eye movements to 8,521 questions
The researchers built the benchmark from egocentric videos with recorded gaze, drawing on datasets including EGTEA, EgoExoLearn, and HoloAssist. Their semi-automatic pipeline converts raw gaze trajectories into questions tied to specific places and times in each video.
- Extract fixations. The pipeline identifies moments when the wearer’s gaze settles on a location.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
