Skip to main content
Back

Kyutai's AURA-1.3B Streams a Live Talking Avatar 5x Faster Than Rivals

Kyutai's AURA puts a lip-synced face on live voice AI, streaming 576p talking-head video at 20.7 fps on a single H100.

  • Kyutai open-sourced AURA, a real-time audio-driven talking-head video model.
  • AURA-1.3B streams 576p at 20.7 fps on a single H100, faster than playback.
  • AURA-5B streams 720p at 9.7 fps, 17x faster than its TalkVerse teacher.
  • Two-stage distillation turns a 50-step bidirectional teacher into a 4-step causal student.
  • Rolling KV cache keeps memory constant for arbitrarily long streams.
  • Live demo, weights, code and 30K-clip dataset released at aura.kyutai.org.

Kyutai, the French open-science lab behind Moshi and Unmute, has released AURA, an audio-driven model that turns a portrait and live speech into a lip-synced talking-head video. The model generates successive video chunks as audio arrives, and Kyutai has published the weights, inference code, paper, research dataset, and an interactive demo.

AURA moves audio-driven video closer to interactive use. Earlier diffusion systems typically process multi-second clips through dozens of denoising passes and may depend on future audio. AURA-1.3B uses four passes, conditions only on audio received so far, and generates frames faster than playback on a single Nvidia H100.

Two sizes, sharply different speeds

Kyutai’s reported performance on one H100
Model Parameters Resolution Denoising steps Throughput Primary role
AURA-1.3B 1.3 billion 576p 4 20.7 fps Interactive demo
AURA-5B 5 billion 720p 4 9.7 fps Higher-resolution generation and research benchmarks

The 1.3-billion-parameter model generates 576p video at 20.7 frames per second, fast enough to exceed its playback rate. Kyutai uses this version in the browser demo. The 5-billion-parameter model raises the resolution to 720p while reducing throughput to 9.7 fps, so it streams incrementally but falls below real-time playback.

Both figures come from an H100, a data-center GPU. Consumer-GPU performance remains unreported in the release materials, so the phrase “single GPU” should be read in that hardware context.

The delay inside audio-driven diffusion

Most audio-driven video diffusion models generate convincing faces by repeatedly removing noise from a complete video segment. That process often requires 40 to 50 denoising passes over several seconds of material, which increases computation and delays the first usable output.

Interactive systems also require causal generation, meaning each chunk can depend on the portrait, prior frames, and audio received so far. A model that uses future audio must wait for that audio to arrive before rendering, preventing a continuous conversational response. AURA’s technical site describes how Kyutai removed that dependency.

Four steps and a two-chunk memory

Kyutai trains AURA as a causal student of TalkVerse, a bidirectional 5-billion-parameter teacher that uses 50 denoising steps per chunk. The student reduces that process to four steps through two stages:

  1. Trajectory regression. The frozen teacher traces a path from random noise to a clean video, commonly described as an ODE trajectory. The student learns to reach corresponding points on that path with far fewer updates.
  2. Backward-simulation distribution matching. The student generates its own trajectory without gradient tracking. Training then aligns teacher and critic scores on the states the student actually visits, correcting errors introduced by the four-step shortcut.

Three architectural changes make the distilled model suitable for streaming:

  • Block-wise causal attention processes chunks of four latent frames, each representing about 0.64 seconds, while restricting attention to available context.
  • Cached temporal context replaces separate motion-frame conditioning. The key-value cache carries recent motion information between chunks.
  • Mixed diffusion timesteps assign timesteps per chunk, allowing past chunks to be cached once instead of recomputed during every denoising pass.

The rolling key-value cache retains two pieces of history: the first chunk as a fixed identity anchor and the most recent chunk for continuity. Its memory requirement therefore stays constant as the generated video grows.

A fivefold lead over the nearest baseline

Kyutai evaluated AURA-5B on the HALLO3 and CelebV-HQ benchmarks. In the reported speed comparison, its 9.7 fps throughput is 5.1 times Live Avatar’s 1.9 fps, 16.7 times the TalkVerse teacher’s 0.58 fps, and more than 120 times Wan-S2V’s 0.08 fps.

Headline results from Kyutai’s baseline comparison
Model Parameters Resolution Steps Throughput CSIM
Hallo3 14.5B 480p 50 0.22 fps 0.74
Wan-S2V 14B 720p 40 0.08 fps 0.71
TalkVerse teacher 5B 720p 50 0.58 fps 0.74
Live Avatar 14B 480p 4 1.9 fps 0.77
AURA-5B 5B 720p 4 9.7 fps 0.79

CSIM measures how closely the generated face preserves the reference identity, with higher scores indicating a closer match. AURA records the highest CSIM among the compared systems on both test sets, beats its TalkVerse teacher on most reported quality metrics, and performs similarly to the much larger Wan-S2V on Fréchet Video Distance.

A human study with 30 raters and 300 pairwise comparisons also favored AURA over Hallo3 for visual quality, lip synchronization, reference fidelity, and naturalness.

Throughput leaves a latency budget

The 20.7 fps result measures generation throughput rather than complete conversational latency. Time to first frame depends on chunk formation, model execution, video decoding, and display. A deployed assistant also adds microphone buffering, speech processing, network transport, and any language or speech-generation stages.

Each latent chunk represents about 0.64 seconds of video, which provides useful context for interpreting the frame-rate result. AURA removes audio look-ahead and generates quickly enough to sustain playback, while end-to-end response time remains a system-level measurement.

Quality frays below four steps

Hallo3 still leads AURA on FID and JEDi, two distribution-based quality metrics, across both test sets. Kyutai’s qualitative review found exaggerated mouth motion and recurring hand artifacts in Hallo3 outputs that those aggregate scores did not capture, illustrating the limits of relying on a single benchmark family.

Reducing AURA below four denoising steps causes a rapid quality decline. At two steps, JEDi degrades modestly. At one step, FVD rises from 195 to 229 and identity preservation falls substantially.

Longer sequences expose another limitation. In a 60-second stability test, CSIM declines from 0.76 during the first 10 seconds to 0.69 by the one-minute mark, indicating gradual identity drift.

From browser demo to research baseline

Kyutai has connected AURA-1.3B to Unmute, its cascaded voice AI stack, allowing users to speak with a generated, lip-synced avatar in the browser. The same architecture could support several latency-sensitive applications:

  • Voice assistants with animated faces
  • Virtual tutors and live presenters
  • Game characters driven by streaming dialogue
  • Customer-service agents that respond without prerecorded clips
  • Experiments in causal, low-step video diffusion

Researchers receive the model weights, inference code, a BMVC paper, and a dataset of 30,000 synthetic clips with precomputed teacher trajectories. Those materials expose both the deployed model and the intermediate supervision used for distillation.

Systems that accept arbitrary portraits also carry impersonation and consent risks. Production deployments need controls around uploaded identities, disclosure of synthetic output, provenance, and misuse reporting alongside the generation stack.

AURA demonstrates that audio-driven diffusion can operate causally with four denoising steps and a bounded attention cache. Its reported results give developers a concrete architecture for interactive avatars, along with clear constraints in accelerator requirements, end-to-end latency, low-step quality, and long-run identity stability.