Runway Slashes Characters Avatar Cost by Running on Half the GPUs
Runway now runs its 57B-parameter Characters video model at 24 fps on four H100s instead of eight, mostly by deleting work it was redoing hundreds of times per step.
- Runway now serves its 57B-parameter Characters video model at 24 fps on four H100s, down from eight.
- Moving RoPE, FP8 quantization, and window shifting to cache write time cut cache upkeep from 69.7ms to 6.2ms per step.
- A fused FP8 GEMM plus all-reduce kernel hides 54-68% of GPU communication, bringing the step to 147.4ms (1.63x).
- Lip-sync quality stayed within run-to-run noise; in production, 99.5% of sessions averaged above 23.5 fps over seven weeks.
- Production bugs included a prompt-length memory leak and one .item() call that added roughly 110ms per step. Details in the paper.
- The API costs about $0.20/minute; the voice agent now takes 1.42s of a 2.13s median conversational turn.
Runway has cut the accelerator requirement for Runway Characters from eight H100 GPUs to four per conversation while sustaining 24 frames per second. The product turns a source image into an interactive character with synchronized speech and video. Its roughly 57-billion-parameter autoregressive video diffusion transformer generates four-frame chunks, with each chunk conditioned on the preceding video.
Because each conversation runs as a private batch-size-one stream, halving the GPU count directly affects serving capacity and cost. Runway’s research paper attributes the gain to three changes: eliminating repeated key-value cache work, fusing GPU communication with matrix multiplication, and overlapping video decoding with the next generation step. Production deployment then exposed memory, compiler, and synchronization failures that fixed-input benchmarks had missed.
Four frames, 166.7 milliseconds
At 24 fps, four frames represent 166.7 milliseconds of video. Characters must finish each four-frame generation step within that window for the duration of a conversation. A late chunk freezes the character and stalls its audio because each frame carries the corresponding slice of speech.
Batching offers little help in this workload. Every session has its own context and timing, while one stream requires 2.24 PFLOP/s of useful arithmetic, exceeding a single H100’s dense FP8 peak. Adding another user to the same GPUs nearly doubles step latency.
Runway’s launch software took 240.9 milliseconds per step when benchmarked on four H100s, missing the deadline by 74.2 milliseconds. The initial deployment therefore reserved eight H100s for each conversation.
Cache upkeep consumed 29% of each step
A representative 239.2-millisecond profile showed that cache maintenance consumed 69.7 milliseconds. Each generation step performs four denoising iterations through a 60-block transformer, producing 240 attention calls. Every call accessed its block’s 5,788-slot key-value cache, even though 757 slots contained newly generated data.
The original implementation repeatedly applied rotary position embeddings, or RoPE, to cached keys, converted the cache to eight-bit floating point, or FP8, and shifted the sliding window by copying memory. RoPE encodes token order by rotating key vectors according to their positions. Attention itself consumed 6% of GPU time; preparing its cache cost far more.
The redesigned cache performs each transformation when data enters the cache:
- Write-time RoPE and FP8 conversion: Each new frame is rotated and quantized once, using per-head scales calibrated for the session.
- Ring-buffer storage: New frames overwrite the oldest slots, advancing the window without copying existing entries. Cache maintenance fell from 69.7 to 6.2 milliseconds per step.
- Delta rotation: Model positions reset every 42 generation steps. At each reset, the cached window receives one corrective rotation. That reset step takes 263 milliseconds, and a playout buffer of ready frames absorbs the delay.
Correct positional encoding depends on the ring-buffer layout. After the buffer wraps, physical slot order no longer matches temporal order. Applying RoPE at write time preserves each frame’s true position, while the attention kernel reads the ring layout directly. The key side of the pattern can be reduced to the following pseudocode:
class RingKeyCache:
def __init__(self, slot_count):
self.keys = alloc(slot_count)
self.ptr = 0
def write(self, new_keys, positions, scale):
rotated = apply_rope(new_keys, positions)
packed = to_fp8(rotated, scale)
offsets = arange(len(packed))
indices = (self.ptr + offsets) % len(self.keys)
self.keys[indices] = packed
self.ptr = (self.ptr + len(packed)) % len(self.keys)
def view(self):
# The attention kernel consumes the ring layout directly.
return self.keys, self.ptrPut the all-reduce inside the matmul
With cache maintenance reduced, tensor-parallel communication became the largest cost after matrix multiplication. Tensor parallelism splits each transformer block across GPUs. After every block, an all-reduce operation sums the partial outputs, producing 240 reductions for each generated chunk.
Replacing the communication library saved 0.4 milliseconds because every reduction still waited for the full matrix multiplication to finish. Runway instead extended ParallelKittens’ fused general matrix multiplication and all-reduce kernel to support FP8 and odd GPU counts. Each completed output tile enters the NVSwitch reduction while later tiles are still being computed. The fused call takes 153.8 microseconds instead of 208.2 microseconds, hiding 54–68% of communication time.
The optimized four-H100 layout assigns three GPUs to tensor-parallel diffusion and one to the variational autoencoder, or VAE, decoder that converts the model’s latent representation into pixels. Decoding takes 114 milliseconds per chunk. The dedicated GPU decodes the previous chunk while the other three generate the next one, preventing the decoder from stalling the tensor-parallel group.
Four H100s clear the deadline
The optimized configurations all bring the four-frame generation step below the 166.7-millisecond real-time budget:
| Configuration | Step latency | Throughput or result |
|---|---|---|
| Launch code, 4 × H100 | 240.9 ms | Misses the real-time deadline |
| Optimized without fused kernel, 4 × H100 | 159.3 ms | 24.5 fps sustained, 1.51× faster |
| Optimized with fused kernel, 4 × H100 | 147.4 ms | 1.63× faster, reaching 43% of FP8 peak |
| Optimized, 3 × B200 | 154.3 ms | Meets the real-time deadline |
| Optimized, 8 × H100 | 118.2 ms | Decoder limits output to 31 fps |
Seven weeks of production telemetry showed average worker throughput above 23.5 fps in 99.5% of sessions that streamed for at least one second. In matched four-pass tests, the paper’s two open-source comparison systems either missed real time or added more than one second of latency, despite generating at one-third to one-half of Characters’ resolution. One of those latency figures was estimated.
Quality tests follow lip sync
Three optimizations alter the model’s arithmetic: the FP8 cache, fused pre-attention kernels, and fused all-reduce. Their errors can compound because each generated chunk becomes context for the next one.
Pixel-level comparisons lose diagnostic value in an autoregressive video stream. Small numeric differences can send two runs down different generation paths, even when both outputs remain valid. After about 10 seconds, repeated runs of the same production configuration differ in peak signal-to-noise ratio, or PSNR, as much as runs using different configurations.
Runway evaluated lip-sync performance with SyncNet, a model that scores alignment between speech and mouth movement. The 53-clip test set covered photorealistic, cartoon, claymation, fantasy, and non-human characters. Results from each optimization were compared with the natural score variation between identical runs.
SyncNet produced scores for all runs on 51 clips. Mean differences were at most 0.03, and every 95% interval stayed within ±0.08. The experiment bounds the observed effect without proving equivalence, and SyncNet is less reliable on stylized characters. The evaluation method applies to other streaming generative systems in which small numerical changes accumulate over time.
Production found the benchmark gaps
Production traffic exposed three failures that fixed prompts, eager execution, and isolated performance tests had not triggered:
- Prompt-length memory leak: After PyTorch and NVIDIA Collective Communications Library upgrades, the fused kernel retained a separate buffer for every first-pass token count. Each previously unseen prompt length allocated more memory. Fixed-length benchmarks never exercised that path. Configuration flags around each optimization allowed Runway to roll back the kernel without redeploying the service.
- Compiler-dependent buffer corruption: One compiled path legally moved a call’s read past the following call’s write to a shared buffer. The eager path passed, while the compiled path left 96% of the buffer’s elements incorrect.
-
A synchronizing
.item()call: A monitoring change extracted an FP8 scale from the GPU, forcing the CPU to wait for GPU completion. Step latency rose from 168.5 to 278 milliseconds because the CPU could no longer prepare later work while the GPU ran.
Kernel availability also changes the value of an optimization across GPU generations. The FP8 cache saves 17 milliseconds on H100 but adds 2.4 to 6.7 milliseconds on B200. Runway’s B200 build lacks an FP8 attention kernel that accepts per-head scales, so the same cache format follows a slower path.
Voice takes two-thirds of turn latency
Across traced live sessions, the median interval from the end of a user’s speech to the character’s audio reaching the client was 2.13 seconds. The voice agent, including the language and speech pipeline, accounted for 1.42 seconds. Runway’s video pipeline contributed 0.49 seconds, leaving about 0.22 seconds for networking and other stages.
The paper cites an average human conversational response gap of roughly 208 milliseconds. Under the optimized configuration, further reductions in end-to-end response time depend primarily on the language and speech components.
The API costs about 20 cents a minute
Developers can access Characters through Runway’s API under the model ID gwm1_avatars, as well as through Runway’s applications. Each active session starts with a two-credit charge, followed by two credits for every six seconds of active time.
At the published price of $0.01 per credit, usage costs about $0.20 per active minute plus the initial $0.02 charge. The product supports tool calling, a knowledge base, and an embeddable web widget for applications such as customer support, tutoring, and branded characters.
A third-party pricing comparison reports a five-minute session cap, so teams planning longer conversations need to verify the current limit with Runway. The model weights and serving implementation remain proprietary.