LivingScreen Exposes How 13 Top AI Models Fail at Scrolling TikTok
LivingScreen exposes a blind spot in every frontier AI agent: none can decide how long to watch a video, and all fall far short of human performance on TikTok-style feeds.
Takeaways
- LivingScreen is the first benchmark for GUI agents operating on auto-playing short-video feeds, with 499 tasks across 1,528 videos. (paper)
- No frontier model reaches human performance: the best model (Gemini-3.5-Flash) scores 69.3% vs. 94.0% for humans, while humans also watch far less video.
- The dominant failure mode is over- and under-observation: models watch either too much irrelevant content or too little of what matters, affecting 39-61% of trajectories.
- Humans spend 52% of observations in a quick "glance" mode to scout relevance; no model exceeds 26% in that category, revealing a missing information-foraging capability.
- Prompt engineering (Watch-Less, Watch-More, Mimic-Human) shifts observation behavior but does not improve accuracy, confirming this is a fundamental capability gap, not an awareness issue.
- Code and data will be released at github.com/BITHLP/LivingScreen; the environment uses Playwright on a browser-based short-video replica with automatic grading.
Every GUI agent benchmark you have ever seen makes a quiet assumption: the screen freezes between actions. Click a button, get a screenshot, decide what to do next. That loop works fine for filling out forms or booking flights. It completely breaks down the moment a short video starts auto-playing. GUI agents today assume a static screen, where the world is frozen between two actions. Real interfaces such as short-video applications violate this assumption, as their content keeps playing, and a competent user must decide what to watch and for how long.
A team from Beijing Institute of Technology, Tsinghua, and Beihang University has now put a name to this gap and built the first benchmark to measure it. The result is LivingScreen, a paper that tests 13 frontier models on short-video platforms and finds that not one of them comes close to human performance. The dominant failure mode is something they call over- and under-observation: models systematically watch either too much or too little, and no amount of prompt engineering fixes it.
The gap nobody was measuring
The GUI agent field has produced impressive benchmarks for the web (WebArena), mobile (AndroidWorld), and desktop (OSWorld). Benchmarks for video understanding have grown rapidly in scale and difficulty, but they universally adopt a passive consumption protocol, where fixed clips and a question are handed to the model. Neither category captures what happens when a video is playing live inside the interface the agent is also trying to operate.

The authors frame this as a two-axis problem. One axis is whether the screen evolves on its own (static vs. living). The other is whether the agent proactively chooses what to look at (passive vs. proactive). The upper-right quadrant, which represents a living screen observed proactively, becomes a gap and the work's target. Short-video platforms naturally live in that quadrant. Every existing benchmark lives somewhere else.
What makes a screen "living"
The paper introduces a formal model to describe the difference. In a standard GUI agent setup, the world only changes when the agent acts. In a living-screen setting, the video playhead advances whether the agent does anything or not. The authors capture this with two modifications to the standard decision-making framework (a POMDP, or partially-observed Markov decision process):
- Continuous evolving state: The screen state changes autonomously between agent decisions, not just in response to them. The video keeps playing.