JD.com's JoyAI-Video-Edit Rewrites Live Video Streams at 30 FPS

JD.com's open source arm has released JoyAI-Video-Edit, a system that edits video streams frame-by-frame as they arrive rather than batching an entire clip through a diffusion model. Point it at a webcam or an uploaded file, type an instruction like "turn this into a watercolor wash," and it rewrites frames on the fly. The repo ships under Apache 2.0 with checkpoints on Hugging Face.

The end-to-end pipeline reaches 30.19 FPS at 720x1280 in the deployment benchmark, which is what makes this worth paying attention to. Most instruction-guided video editors are offline pipelines that require the entire clip up front, run for minutes per second of output, and revisit future frames to maintain temporal coherence. JoyAI-Video-Edit is causal: it never peeks ahead.

What's actually under the hood

The system combines an MLLM-based condition encoder, a causal video VAE, and a 16B-parameter multimodal diffusion transformer. The MLLM (a multimodal LLM) reads a text instruction plus the current video context and produces the conditioning signal. The causal VAE compresses frames into latents without looking at future frames, and the 16B MMDiT backbone denoises each chunk.