GEST-Engine Beats VEO 3.1 on Multi-Actor Video Using GTA San Andreas
A new game-engine pipeline outputs multi-actor videos with per-frame 3D annotations, exposing that VLM encoders miss basic spatial structure.
ProTakeaways
- GEST-Engine generates multi-actor videos on GTA San Andreas with per-frame 3D spatial relation graphs and event-to-frame mappings, paper here.
- GTASA dataset: 938 videos, 28.2K action clips, 892M pairwise spatial relations, 29.6K event-frame mappings, released open-source.
- Human eval: 69% physical validity vs 18% for VEO 3.1 and 13% for WAN 2.2 on identical prompts.
- Two-stage fine-tuning with GEST data beats VEO-pretraining and real-only training on VideoLLaMA3 2B and Qwen3-VL 2B captioning.
- Probing 4 frozen encoders on 11 tasks shows V-JEPA 2 and VideoMAE V2 significantly outperform Qwen3-VL visual encoder on spatial reasoning.
- Frontier VLMs (GPT-5.2 Pro, Gemini 3.1, Claude Opus 4.6) hit only ~56% accuracy detecting physically invalid videos.
State-of-the-art video generators can render photorealistic clips that fall apart the moment you count actors, track objects through time, or check whether a scene obeys physics. A new paper argues that if you want ground truth for what is happening in a multi-actor video, you should stop trying to squeeze it out of neural generators and instead borrow the explicit world model that already lives inside a 3D game engine.
The system is called GEST-Engine, and the accompanying corpus, GTASA, is built on top of GTA San Andreas via Multi Theft Auto. It produces synchronized RGB video, per-frame pairwise spatial relation graphs, event-to-frame temporal mappings, and matching text descriptions, all as a free by-product of running the simulation.
Why neural generators keep failing the physics check
The authors frame the core issue clearly: video understanding models still struggle with fine-grained temporal reasoning, and video generators exhibit spontaneous appearance and disappearance of objects alongside physically implausible interactions. Existing datasets don't help you diagnose this because they trade off scale for annotation richness. Large-scale collections provide millions of clips but only text descriptions and coarse temporal labels, while densely annotated ones like Action Genome or PVSG rely on expensive human labeling with documented noise.
Synthetic platforms exist, but they've been narrow. CARLA is driving-only, BEDLAM is single-actor motion capture, VirtualHome tracks object state changes without pairwise spatial graphs. None handle multi-actor narrative coordination with dense annotations in one framework.
How GEST-Engine builds a scene
The pipeline starts with a Graph of Events in Space and Time (GEST), a specification of actors, actions, objects, and their temporal and spatial constraints. A procedural generator reads the engine's capability registry so every generated graph is executable by construction. The workflow runs in five stages: