HKU's ViMax Beats Sora and Veo With Multi-Agent Film Crew Approach
ViMax turns any idea, script, or novel into a multi-scene video by coordinating a team of specialized AI agents — solving the consistency and length limits that have plagued AI video generation.
Takeaways
- What it is: ViMax is an open-source multi-agent framework from HKU that turns ideas, scripts, or novels into multi-scene videos with consistent characters.
- How it works: Four specialized agents (Screenwriter, Director, Producer, Video Generator) coordinate like a film crew, using RAG, dependency graphs, and VLM quality control.
- Key innovation: A graph-based visual dependency system tracks character and environment state across shots; removing it drops global consistency by 8.7%.
- Benchmarks: ViMax wins 74-80% of human preference comparisons against Veo 3.1, StoryDiffusion, and IC-LoRA on multi-shot consistency and narrative quality.
- Limitations: Depends on commercial APIs (Google Veo, Nanobana) for pixel generation; no tagged releases yet; no audio/dialogue sync support.
- Access: MIT licensed, free to use; API costs for Google Veo and Nanobana apply. Install with
uv syncafter cloning.
ViMax, an open-source framework from the University of Hong Kong's Data Science Lab, takes a fundamentally different approach to AI video generation. Instead of asking a single model to conjure a video from a prompt, it deploys a coordinated team of specialized agents , a Screenwriter, Director, Producer, and Video Generator , that collaborate like an actual film crew. The result is multi-scene, minute-plus videos with consistent characters, coherent narrative, and synchronized audio, all from a single idea or novel.
The problem every AI video tool shares
Current AI video generation tools suffer from a short clip syndrome , most tools produce only 2-5 second clips, not minute-long or hour-long content , and consistency chaos, where characters and scenes change unpredictably across frames. Models like Sora, Runway, and Veo are impressive at the shot level, but they have no concept of "the same character from scene 1." The moment you need scene 2 to follow logically from scene 1, the single-prompt pipeline breaks down.
Current video generation methods face critical limitations, producing only short clips without narrative structure while character appearances and environmental elements change unpredictably across scenes. ViMax attacks this at the architectural level, treating video production as a multi-agent orchestration problem rather than a one-shot generation problem.
Four agents, one film crew
Unlike most AI video tools that generate only a few seconds of footage, ViMax orchestrates 12 specialized agents that collaborate like a real film production team: a Screenwriter, Storyboard Artist, Character Extractor, Reference Image Selector, Best Image Selector, and more , all coordinated by a central orchestration layer. At a higher level, these collapse into four roles:
- Screenwriter: Takes a raw idea, novel, or existing script and produces a fully structured screenplay with characters, scene segmentation, dialogue, and transitions. A RAG-based long script design engine intelligently analyzes lengthy, novel-like stories and automatically segments them into a multi-scene script format, meticulously ensuring that all key plot developments and character dialogues are accurately retained within the new structure.
- Director: A shot-level storyboard design system creates expressive storyboards through cinematography language based on user requirements and target audiences, establishing the narrative rhythm for subsequent video generation.
- Producer: The consistency engine. Selects reference images, validates that the same character looks the same across shots, and runs multimodal LLM (MLLM) consistency checks. This is the layer that solves the character-reshuffling problem.