Video-Index Exposes 35 Video Benchmarks AI Can Ace Without Watching
A five-level shortcut audit cracks 77 of 115 video benchmarks, then distills 505,518 questions down to 840 that actually require watching.
- Audit of 115 video benchmarks finds 35 can be solved with zero frames seen.
- Shuffled frames preserve 96% of full-video accuracy on 51 temporal benchmarks.
- Near-duplicates make up at least half the items in 63 benchmarks.
- Video-Index curates 840 shortcut-resistant questions from 76 sources, four capability groups.
- Claude Opus 5 beats every open-source model by 37+ points under fixed input.
- Code, data, and pipeline released; agents hit 79.3% while humans score 55.0%.
Video-Index finds shortcuts in 115 video benchmarks
An audit of 115 popular video benchmarks found that 35 can approach full-video accuracy without inspecting a single frame. The team behind Video-Index tested each benchmark with five increasingly capable shortcut attacks, then used the results to curate 840 questions that still require visual evidence.
Benchmark aging drives the problem. Each test was designed around the models available at the time, while newer systems have stronger language priors and can extract more information from a single image. A question that required video evidence in 2023 may yield its answer through wording, dataset patterns, or one representative frame in 2026.
Five shortcuts expose benchmark decay
Video-Index defines an attacker as any strategy that raises accuracy without using the capability a benchmark claims to measure. Its “attack pyramid” grants progressively more access to each evaluation item:
| Level | Available information | Shortcut being tested |
|---|---|---|
| 1 | Answer options only | Whether the choices reveal the likely answer |
| 2 | Question text and options | Whether language alone is sufficient |
| 3 | Other items in the evaluation pool | Whether templates, repetition, or dataset patterns expose answers |
| 4 | One frame or a caption | Whether static visual evidence solves the task |
| 5 | Shuffled frames or a short contiguous clip | Whether temporal order or broad video coverage is necessary |
The audit uses Gemini 3.5 Flash with 32 sampled frames as its full-video reference. A shortcut breaks a benchmark when its above-chance score comes within five percentage points of that reference. Benchmarks that remain more than five points ahead of every shortcut survive the audit.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
