Skip to main content
Back

OmniScript's 8B Model Beats Gemini 3-Pro at Reading Full Movies

A new research paper defines the video-to-script task, releases a benchmark, and shows an 8B model matching Gemini 3-Pro on cinematic video understanding.

Pro

  • New arXiv paper introduces Video-to-Script (V2S), a task for generating hierarchical, timestamped scripts from long cinematic videos.
  • Authors release OmniScript, an 8B omni-modal model built on Qwen3-VL with audio-visual DeepStack injection.
  • Training uses chain-of-thought SFT plus GRPO reinforcement learning with temporally segmented event-level rewards.
  • Benchmark spans 10 films, 16.8K events, 19.9 hours, evaluated with hierarchical metrics and temporal IoU.
  • Matches Gemini 3-Pro overall, beats it on dialogue (72.2 vs 68.8) and temporal localization (69.3 vs 64.4).
  • Code, weights, and demo listed as coming soon on the project page.

Turning a two-hour movie into a structured, timestamped screenplay is still something today's video models mostly cannot do. Short clips work fine. Full cinematic content with recurring characters, overlapping dialogue, ambient sound, and shifting scenes tends to break current multimodal LLMs. A new arXiv paper called OmniScript takes a direct swing at this gap, introducing a formal task, a human-annotated benchmark, and an 8B model that reportedly holds its own against Gemini 3-Pro.

Authored by Junfu Pu, Yuxin Chen, Teng Wang, and Ying Shan, the paper defines a task, builds evaluation tooling for it, and releases a model trained to solve it. The proposed video-to-script (V2S) task aims to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To support it, the authors construct a human-annotated benchmark, propose a temporally-aware hierarchical evaluation framework, and present OmniScript itself: an 8B-parameter omni-modal language model tailored for long-form narrative comprehension.

Why plain captioning falls short

Most video captioning systems produce one paragraph per clip. That works for TikTok-length content, but a cinematic scene contains dozens of intertwined signals: who is speaking, what they are doing, how they look while doing it, what music is playing, when the camera cuts to a new location. Flattening all of that into a single caption throws away the structure that makes the content interesting in the first place.

V2S formalizes the output as a three-level hierarchy. A meta level covers title, duration, and the full character list. Underneath sits a sequence of scenes tagged with location, environment, time of day, and mood. Inside each scene sits a list of timestamped events, each carrying a character identity, dialogue, action, expression, and audio cue. The result resembles an actual screenplay rather than a caption dump.