Code2Video Bench tests LLM-written motion graphics

HeyGen Research and Kaggle have launched Code2Video Bench, a benchmark that measures how well AI agents turn creative briefs into motion-graphics code. The approach gives production teams editable source files, repeatable renders, and precise control over timing, typography, assets, and animation.

Current models remain far from the human references. In HeyGen’s human-reference comparison, almost every model scored near zero. Because those scores offered little separation, the public leaderboard uses pairwise Elo ratings. An Elo rating summarizes head-to-head results and has no standalone quality interpretation.

Brief in, reproducible MP4 out

1. Brief: The model receives a production-style prompt specifying composition, typography, colors, timing, transitions, and motion. One example requests a 6.8-second Figma-style product hook that zooms out from a blooming vector flower into a three-card page and live timeline editor.
2. Code and render: The agent writes HTML, CSS, and JavaScript. The open-source Hyperframes engine renders that code into output.mp4 inside an isolated Kaggle Harbor sandbox. Deterministic rendering means the same code and environment produce the same frames.
3. Pairwise evaluation: HeyGen’s in-house judge compares two candidate videos and estimates which one a human would prefer. It assesses prompt intent, composition, temporal rhythm, craft, and motion separately, preserving axis-level diagnostics alongside the overall Elo ranking.

Code2Video contains 168 human-created briefs covering common product-launch sequences: hooks, product introductions, feature demonstrations, social proof, and calls to action.

The leaders are bunched together

The reported confidence intervals overlap near the top of the leaderboard, limiting the significance of small rank differences.

Open-weight Qwen 3.8 Max trails the leading proprietary model by 35.2 Elo points. Across the broader leaderboard, more than 165 points separate first and 16th place. The axis-level results explain most of that spread.

Intent survives; motion breaks

Prompt intent is the strongest and most tightly clustered axis, indicating that leading models usually include the requested scenes and elements. Composition, temporal rhythm, craft, and motion create more separation. HeyGen also reports regressions in some newer systems: GPT-5.5 and Claude Opus 4.8 sometimes outperform their successors when newer models add unnecessary layout complexity.

• Broken timing: New elements begin moving before earlier animations settle, producing crowded transitions even when individual cues start at the requested times.
• Mid-flight collisions: Models arrange text for its final position without checking the intermediate frames, allowing elements to overlap while moving.
• Incomplete reveals: Typewriter effects can stop one character early. GPT-6-Astra, ranked second, renders “Make it your own” as “Make it your owr” in one example.
• Defensive styling: Uncertain models fall back to small type, thin lines, faint backgrounds, and wide margins. A requested world map covering two-thirds of the canvas may appear as a small centered graphic.

An 82% judge with defined limits

Across 1,464 evaluation samples, HeyGen reports 82 percent agreement between its specialized judge and human preferences. The general-purpose vision-language models it tested reached 75 percent. Agreement exceeded 95 percent on comparisons where the judge expressed stronger confidence.

Those figures apply to videos rendered with Hyperframes and evaluated under HeyGen’s five criteria. The report does not establish equivalent accuracy for other renderers, visual styles, or creative tasks, so teams should validate the judge against their own human ratings before using it as a release gate.

Turn failure modes into guardrails

Code2Video measures capabilities that many image and video evaluations leave unexamined: programmatic timing, animation sequencing, intermediate-frame layout, exact text rendering, and deterministic asset placement. These properties matter for ads, product demos, and explainers that must remain editable after generation.

The benchmark results suggest several practical checks for production agents:

• Validate timeline dependencies so transitions wait for preceding motion to settle.
• Test collisions across intermediate frames, not only at final positions.
• Assert exact on-screen text after typewriter and reveal effects complete.
• Set minimum size and canvas-coverage rules for primary visual elements.
• Use axis-level critique and human review to catch composition and motion defects.

Teams choosing among the highest-ranked models should treat the leaderboard as a comparative signal, since overlapping intervals and narrow Elo gaps limit what rank alone can establish. The Kaggle leaderboard includes tasks and side-by-side failure clips, while the GitHub repository provides the renderer for local experiments.