Meta cuts real-time avatar generation from 120 model evaluations to two

Meta has unveiled Muse Realtime Avatar, a video-generation system that turns a reference image into an animated character for live conversation. It accepts portraits, full-body illustrations, animals, and objects, then generates synchronized speech, lip movement, expressions, and gestures at 25 frames per second.

Meta introduced the system in a Connect announcement and plans to integrate it with the Muse app. It enters the same market as Runway Characters and HeyGen LiveAvatar. Its main technical contribution is a method for running a large diffusion video model at interactive speeds while limiting visual drift during long sessions.

One speech stream drives both outputs

Muse Realtime Voice produces vector-quantized speech tokens, discrete codes that represent both linguistic content and delivery. An audio decoder converts those tokens into speech, while the avatar model consumes the same timed stream to generate matching video. Shared conditioning keeps lip movement, cadence, and expression aligned without a separate post-processing step.

The video generator is an audio-driven diffusion transformer conditioned on three inputs: the speech tokens, the reference media, and a rolling window of recent video latents. Latents are compact numerical representations of generated frames. The model emits short causal chunks using only past context, then feeds the newest latents into the next chunk. A fixed context window keeps memory and computation bounded as a call continues.

From 120 model calls to two

Meta begins with a bidirectional teacher model that is too expensive for live inference. It runs 40 diffusion steps with three-way classifier-free guidance, requiring three network evaluations per step and 120 evaluations for each video chunk. Classifier-free guidance combines conditioned and unconditioned predictions to improve adherence to the input.

The causal student reduces inference to two unguided evaluations per chunk, cutting neural function evaluations by 60×. Two training methods preserve quality:

• Guidance-aware distribution matching. Training transfers both the teacher’s output distribution and the effect of classifier-free guidance into the student. The student can therefore produce guided results without running guidance during inference.
• Self-forcing. The student trains on context generated by its own earlier predictions. This exposes it to the same accumulated errors it will encounter during deployment and reduces long-session drift in identity, geometry, and motion.

In Meta’s teacher-versus-student evaluation, human raters selected each model at roughly equal rates. The result suggests that the two-evaluation student retains much of the teacher’s perceived quality while meeting the latency requirements of live video.

What one GB200 can deliver

Meta pairs the distilled model with an inference stack designed for continuous sessions. Persistent key-value caches reuse attention state between chunks. Cache-aware routing and latency-sensitive batching group compatible requests, while 4-bit quantization-aware training, fused kernels, and CUDA Graph capture reduce memory traffic and execution overhead.

The 20 ms figure measures raw generation throughput, equivalent to 2.5 ms of model time per frame. The 870 ms figure measures the interval from the end of a user’s turn until the first byte of the synchronized voice-and-video response reaches the user. Those measurements describe different parts of the serving path.

Large leads in Meta’s own test

Meta compared Muse Realtime Avatar with Runway Characters and HeyGen LiveAvatar through each vendor’s live-call product. Raters held conversations lasting two to three minutes with matched avatar identities, then scored visual quality, audiovisual synchronization, character consistency, and mannerisms.

Point estimates favored Muse overall and in every measured category. The mannerism comparison with Runway did not reach statistical significance. Because Meta designed and reported the evaluation, independent testing is needed to establish how the systems compare across identities, conversation lengths, network conditions, and hardware.

A reusable path to interactive diffusion

Common low-latency avatar systems constrain generation through fixed portraits, head-only crops, or limited motion. Muse instead uses a diffusion transformer to generate new frames for full-body movement and gestures. Its combination of guidance-aware distillation and self-forcing offers a practical recipe for teams trying to move diffusion video from offline rendering into interactive applications.

Meta identifies interactive Muse characters as the immediate use case. The same architecture could support several other products:

• customer-support and sales agents;
• interactive tutors and language partners;
• game characters with generated speech and motion;
• virtual presenters for live streams and events.

Hardware and access set the limits

• No public API or model release: developers cannot yet integrate the system or run it on their own infrastructure.
• Muse-only availability: access is tied to Meta’s Muse app, which is restricted to users aged 18 and older.
• High-end serving hardware: the reported latency and concurrency figures use an NVIDIA GB200. Meta provides no comparable results for common data-center or consumer GPUs.
• Embedded provenance: every generated frame carries an invisible Meta Video Seal watermark.