Meta's Muse Turns Any Portrait Into a Real-Time Talking Avatar

Meta has begun rolling out Muse Realtime Avatar to US adults, adding a visible character to the Muse app’s Realtime Voice assistant. Users can supply a portrait, full-body illustration, animal, or object as a reference image. The system animates it while the assistant speaks, synchronizing lip movement, facial expression, and body gestures with the generated voice.

Meta reports output at 448 × 768 pixels and 25 frames per second, with approximately 870 ms between the end of a user’s turn and receipt of the first voice-and-video response data. The metric captures response startup. Meta’s announcement does not provide full-response duration or results from independent network tests.

One token stream aligns speech and motion

Muse Realtime Voice emits vector-quantized speech tokens, compact codes that capture both the words and their delivery. An audio decoder converts those tokens into speech, while the avatar model uses the same stream to generate matching movement. The shared representation coordinates voice, lip motion, and expression from the start of generation.

The avatar generator is an audio-driven diffusion transformer conditioned on the speech tokens, reference image, and a rolling window of video latents, which are compressed representations of recent frames. It emits short causal chunks that use prior frames as motion context. Feeding each chunk into the next helps preserve the character’s appearance and mannerisms while keeping memory and computation bounded during long conversations.

Two steps replace 120 model passes

A conventional diffusion video model refines noise into video over many denoising steps, making real-time generation expensive. Meta’s teacher model uses 40 steps with three-way classifier-free guidance, for 120 model evaluations per chunk. Classifier-free guidance combines conditioned and unconditioned predictions to steer output quality, but each additional prediction increases inference cost.

The distilled student runs two unguided denoising steps and learns both the diffusion process and the effect of guidance. Meta also uses self-forcing, which trains the student on its own generated context. That exposure helps it limit drift as small errors accumulate across successive chunks. Meta reports a 60-fold reduction in computation per chunk, with human preference close to the slower teacher’s result.

Meta’s tests put Muse ahead

In Meta’s live evaluation, raters watched matched avatars conduct two- to three-minute, phone-style conversations using Muse, Runway Characters, and HeyGen LiveAvatar. The reported dimensions included lip synchronization, mannerisms, and preservation of the reference character.

The comparison reflects Meta’s test design, selected workloads, and raters. Independent evaluations could establish how the results transfer across different characters, speaking styles, networks, and conversation lengths.

One GB200 handles 12 sessions

Meta reports that a single NVIDIA GB200 can serve 12 concurrent real-time video-generation sessions after applying several inference and routing optimizations:

• Generation speed: Each step produces eight frames, equal to 320 ms of playback, in 20 ms. That averages 2.5 ms of model time per frame.
• Serving capacity: The optimized system delivers eight times the capacity of Meta’s two-step BF16 baseline. BF16 is a 16-bit numeric format commonly used for model inference.
• Serving stack: Persistent key-value caches, cache-aware routing, latency-aware dynamic batching, and 4-bit quantization-aware training contribute to the reported gain.

The 12-session concurrency figure applies specifically to video generation. Persistent key-value caches reuse attention state from earlier chunks, while cache-aware routing sends each session to hardware that already holds its state. Latency-aware batching groups work from multiple sessions without exceeding playback deadlines.

Quantization-aware training simulates 4-bit weights during fine-tuning, allowing the model to adapt to lower numerical precision before deployment. The resulting weights require less memory and data movement than the BF16 baseline.

Access stays inside the Muse app

Developers cannot yet integrate the avatar into their own products because Meta has announced no public API. Current access and output are bounded as follows:

• Availability: The rollout is limited to US users aged 18 and older through the Muse app.
• Output: Video uses a fixed 448 × 768 portrait format at 25 frames per second, targeting phone-sized interactions rather than larger production formats.
• Provenance: Every generated video carries a durable, invisible watermark embedded with Meta Video Seal. Meta says watermarking adds no latency to generation.

For voice-agent developers, Muse provides a concrete architecture for synchronizing generated speech and video with sub-second reported startup latency. Shared speech tokens coordinate the modalities, causal latent windows bound the model’s context, and distillation plus low-precision serving reduce inference cost. Open questions include performance under independent testing, efficiency on accelerators other than the GB200, and whether Meta will expose the system through an API.