Runway Rebuilds Gen-4.5 to Stream Video Frames in Real Time

Runway is betting the future of video generation looks less like a render queue and more like a live stream. In a research post, the company laid out how it is post-training its foundation models to generate video frame-by-frame in real time, rather than making users wait for a finished clip to appear.

Today's video generation is relatively static: you input a prompt, wait a few seconds or minutes, then get a finalized output. Runway argues that this batch-style workflow is the biggest bottleneck in creative iteration, and that collapsing time-to-first-frame changes both the user experience and the unit economics of generative video.

From batch renders to streaming frames

The technical setup builds on Runway's existing base models like Gen-4.5. Each autoregressive step is conditioned on two things: an initial first frame and a caption chosen by the user at the start. From there the model generates frame-by-frame while keeping prior latents in context, and the video and audio decoders run causally, streaming the outputs as latents are produced.

Getting there is a two-stage process Runway calls teacher forcing followed by student forcing. In the first stage, the entire model architecture is converted into a temporally causal, frame-by-frame autoregressive generator. That is notable on its own: the same architecture Runway has been shipping for full-attention, bidirectional generation apparently adapts to causal generation without collapsing quality.

Why distillation is the hard part

Causal generation alone is not fast enough. Flow matching takes many denoising steps to produce a frame, so distribution matching distillation is used to compress each frame down to a few steps, fast enough to stream with good latency. Runway runs this in two phases, and the second one is where most of the gains come from.

• Off-policy distillation: The student causally predicts next states given ground-truth context, while a frozen bidirectional teacher shows the student what a good generation looks like and a critic tracks what the student currently produces. Cheap and stable, but limited.
• On-policy distillation: The student generates a sequence autoregressively during training, with each generated latent becoming the context for the next one, so the model sees its own generated context and learns to correct its own drift instead of amplifying it.

On-policy training matters here because of a quirk specific to video. Language models can correct themselves midstream via text, but a small error in video generation compounds over time as the model builds on each frame sequentially. A slightly wrong hand in frame 10 becomes a melting limb by frame 60.

One practical detail worth flagging for anyone training causal video models: Runway found that a curriculum of increasing sequence length outperforms a fixed sequence length. Starting training with long sequences causes generated trajectories to diverge sharply from the teacher's guidance by the end of the rollout, drifting so far out of distribution that supervision points at an outcome the student will never reach.

Inference becomes the new bottleneck

Real-time generation moves the compute bottleneck from training to inference. Every frame has to leave the model fast enough to keep up with playback, on hardware shared across concurrent sessions. That is a very different serving problem than batch video generation, where a single job can hog a GPU for a minute and no one notices latency variance.

Runway also frames the cost angle bluntly. Cost per output at a given quality bar determines which use cases are viable, and instant generation pushes that bar out further, bringing workloads that weren't previously cost-effective into reach. Faster models use fewer GPU-seconds per clip, which changes what you can afford to build on top of them.

What real-time video unlocks

The post ties this work to Runway's other recent releases, including Solaris and GWM Worlds 2, as part of a broader push into interactive and agentic environments. The interesting downstream targets are less about faster TikTok clips and more about closed-loop systems:

• Gaming and interactive media where generated video needs to react to input within a frame budget.
• Robotics and autonomous vehicle simulation, where evaluating how physical agents act in the world requires environments that generate in real time and respond to edge cases instantly.
• Digital agents doing computer-use tasks in generated environments, tying into Runway's Solaris work.
• Education where content adapts as a learner interacts with it.

The bigger takeaway is that the field is quietly shifting away from treating video generation as an offline rendering problem. If Runway is right that interactive experiences become the dominant use case for generative media, then time-to-first-frame and error accumulation under long rollouts start mattering more than raw single-clip fidelity, and the training recipes designed for bidirectional diffusion will need the kind of causal, on-policy retrofitting described here.