NVIDIA's Dynamo Cuts LLM Engine Crash Recovery from 283 Seconds to 7

When a production LLM engine crashes, the standard recovery path is brutal: reload weights into HBM from storage, recompile kernels, recapture CUDA graphs. For large models that can take minutes, and surviving workers get crushed by displaced traffic in the meantime. NVIDIA just previewed a fix in Dynamo called shadow engine recovery that keeps a pre-warmed backup engine idle on the same GPUs, ready to take over the moment the active one dies.

In a GLM-5.2 benchmark, shadow engine recovery takes 7.3 seconds (1.7 seconds to detect the fault and 5.6 seconds to promote the shadow), compared to the cold restart baseline of 283 seconds. That 39x speedup changes what fault tolerance looks like for large-scale inference.

Why cold restarts hurt so much

Most production LLM crashes are not hardware failures. Production engines commonly experience recoverable software faults, including process crashes, recoverable CUDA errors, and transient collective failures. In these cases, the hardware, drivers, and node remain healthy; only the process holding the corrupted state is lost. So why does recovery still take minutes? Two things get in the way:

• Weights are tied to the engine process. GPU memory is linked to the engine's CUDA context, which is itself bound to the process. When the process exits, the driver releases all resources, including weights already resident in GPU memory.
• Some initialization states are non-transferable. NCCL and torch.distributed communicators bind to the specific running process, and CUDA graphs are fixed to the virtual addresses present during capture.

Every restart therefore repeats the full weight load plus kernel compilation, autotuning, and graph capture from scratch.

Decoupling weights from the process

The core innovation is the GPU Memory Service (GMS), a per-GPU sidecar that owns physical GPU memory on behalf of inference engines. It stays mostly dormant and has no CUDA context of its own; it allocates physical pages, hands out handles to them, and arbitrates which engines may read or write at any time. Engines connect, import handles, and map the underlying pages at virtual addresses in their own CUDA contexts.

GMS is built on the CUDA Virtual Memory Management API, which lets physical allocations and virtual addresses have independent lifetimes. Since physical allocations are reference-counted, they survive as long as any process maintains a mapping. When an engine process dies, its CUDA context is torn down, but the weights stay resident in HBM because GMS still holds a reference. A replacement engine just remaps them.

Integration into inference frameworks is surprisingly light. vLLM, SGLang, and NVIDIA TensorRT-LLM each integrate GMS through a custom torch.cuda.CUDAPluggableAllocator bound to the weight memory pool, so from inside the engine, weights remain ordinary tensors.

What a shadow engine actually holds

A shadow is a full engine process that has already booted, captured its CUDA graphs, established NCCL and NIXL communicators, and imported the shared weight handles. Then it parks itself: releases materializable memory, blocks on a lock, and waits.

A parked shadow therefore retains only its CUDA context, captured graphs, communicators, and weight mappings, with no separate copy of the weights and no KV cache. That is the whole reason two engines can co-reside on the same GPUs without blowing the memory budget. The KV cache, the largest reclaimable chunk, is only materialized once the shadow gets promoted.

Coordination is handled with something refreshingly boring: a POSIX flock on a shared file. When the active process exits due to a shutdown, segfault, or SIGKILL, the kernel reaps its file descriptors, and the shadow acquires the lock to begin serving. Kubernetes liveness probes catch the hung-but-alive case by cascading to a SIGKILL, which trips the same release path.

Benchmarks on GLM-5.2

NVIDIA ran the test on two workers serving GLM-5.2 quantized to NVFP4 on NVIDIA B200 nodes: one worker per node, TP=8, 200K max context, and an FP8 KV cache, with 32K input and 1K output tokens per request at 0.7 req/s. They SIGKILLed one worker after the system reached steady state. The user-visible impact:

The catch, and what is missing

This is a preview feature with real edges. A few things worth knowing before you plan around it:

• Shadow engine recovery addresses common engine process failures but does not cover hardware, node, or multi-node failures, which still rely on standard rescheduling.
• It requires Dynamic Resource Allocation (DRA) on Kubernetes, so the cluster needs Kubernetes 1.34 or newer with DRA enabled and the NVIDIA GPU DRA driver installed.
• Because promoted shadows start with empty KV caches, the post-cutover TTFT sees a slight bump. Persisting KV cache across a promotion is on the roadmap.
• vLLM is the primary supported backend, with SGLang and TensorRT-LLM integrations available through the same allocator interface.

To try it, NVIDIA points to the Shadow Engine Failover docs and a ready-to-run vLLM failover manifest in the ai-dynamo/dynamo repo. For teams running SLA-sensitive inference, this is the kind of infrastructure primitive that quietly shifts what uptime targets are reasonable to promise.