Skip to main content
Back

NVIDIA's Dynamo Snapshot Cuts AI Inference Cold Starts by 21x on Kubernetes

NVIDIA Dynamo Snapshot uses checkpoint/restore to cut Kubernetes inference cold-start times from minutes to under 5 seconds, achieving a 21x speedup on 120B-parameter models.

  • 21x faster cold starts: NVIDIA Dynamo Snapshot reduces Kubernetes inference startup from minutes to under 5 seconds for 120B-parameter models.
  • Checkpoint/restore approach: Uses CRIU (Linux process freeze-and-thaw) and cuda-checkpoint to serialize and restore both CPU and GPU state across nodes.
  • KV cache trick: Unmapping the KV cache before checkpointing shrinks the artifact from ~190 GiB to ~6 GiB for Qwen3-0.6B on a B200 GPU.
  • GPU Memory Service (GMS): Decouples model weights from the CRIU checkpoint, enabling parallel weight and process restoration over GPUDirect Storage or NVLink.
  • Current limits: Experimental, single-GPU vLLM/SGLang only; GMS, multi-GPU, and TensorRT-LLM support are on the roadmap.
  • CRIU optimizations pending: AIO and parallel memfd restore improvements are not yet merged into upstream CRIU and not yet shipped.

Every time a Kubernetes cluster needs to spin up a new inference replica to handle a traffic spike, it pays a steep tax. A cold start means the full sequence a model server must complete before serving any request: pulling the container image, loading model weights into GPU memory, warming up CUDA kernels, compiling CUDA graphs, and registering with the service discovery layer. For large models, that bill can run into minutes. NVIDIA's answer is Dynamo Snapshot, a checkpoint/restore system that skips the entire cold-start sequence and brings a fully warm inference worker back to life in seconds.

The GPU idle problem nobody talks about

In production inference deployments, demand fluctuates over time, requiring inference replicas to scale elastically. Cold-starting inference workloads on Kubernetes can take several minutes. During that time, GPUs are allocated but idle, generating no tokens and serving no requests. That is not just wasted compute -- this delay increases the risk of SLA violations during traffic spikes, as the system cannot scale quickly enough to absorb sudden increases in demand.

NVIDIA Dynamo Snapshot is a checkpoint/restore system for AI inference workloads on Kubernetes. It serializes the full state of a running inference worker -- both GPU-side and CPU-side -- and restores it on the same or a different node, skipping the cold-start sequence entirely. The key insight is that you only need to pay the cold-start cost once. After that, every subsequent scale-out event restores from a frozen snapshot instead of booting from scratch.

Two tools, one frozen worker

A running inference worker has two distinct types of state that both need to be captured. Dynamo Snapshot uses one tool per type: cuda-checkpoint serializes GPU device state (CUDA contexts, streams, device memory, virtual address mappings) into CPU memory of the process owning each CUDA context, using the checkpointing capability of the CUDA driver. CRIU (Checkpoint/Restore in Userspace) walks Linux kernel bookkeeping and serializes the host-side process tree (CPU memory, threads, file descriptors, namespaces) to disk.

CRIU is a freeze-and-thaw mechanism: when a process is restored, execution resumes at the exact instruction where it was checkpointed, completely unaware that any of this happened. The two tools compose cleanly -- checkpoint order is: cuda-checkpoint dumps GPU state to CPU memory first, then CRIU dumps all host-side state to storage. Restore order is: CRIU reconstructs the process tree from storage first, then cuda-checkpoint restores GPU state from what is now in CPU memory onto the new GPUs.

On the Kubernetes side, Dynamo Snapshot is deployed as a privileged DaemonSet called snapshot-agent, installed via a Helm chart. An agent runs on every node and handles checkpoint and restore for runc-managed containers without requiring modifications to runc itself. Each agent operates independently, so checkpoints and restores parallelize naturally across the cluster.

NVIDIA Dynamo Snapshot architecture overview

The quiesce trick that makes it work

There is a subtle problem with checkpointing a live inference worker: it has open TCP connections to the Dynamo control plane that CRIU cannot capture in an established state. The solution is a quiesce/resume hook pattern. Kubernetes readiness probes are defined to checkpoint workers at an optimal state -- after engine initialization but before distributed runtime startup. This ensures checkpoint artifacts remain lightweight while avoiding issues with active TCP connections that cannot be restored.

In practice, the worker writes a signal file after its engine is fully warm, then enters a polling loop waiting for a "restore complete" signal. The checkpoint fires inside that loop. On restore, the worker wakes up inside the same loop, detects the completion signal, and proceeds to register with the control plane as if nothing happened.

Three optimizations that make it fast

Getting checkpoint/restore to actually beat a cold start required solving three separate bottlenecks:

  1. KV cache unmap: KV cache unmap via cuMemUnmap and cuMemRelease reduces checkpoint artifact size from ~190 GiB to ~6 GiB for Qwen3-0.6B on a B200. The virtual address is preserved (since it is baked into CUDA graphs), but the physical memory is freed before checkpointing since the cache is empty anyway at that point.
  2. Parallel memfd restore: Model weights in vLLM and SGLang are stored in pinned CPU shadow buffers backed by shared anonymous memory (memfds). Upstream CRIU restores these serially. NVIDIA modified CRIU to launch a thread pool and restore all these buffers in parallel, saturating available storage bandwidth instead of processing them one at a time.
  3. Linux native AIO: Private memory reads are now processed in parallel using Linux native AIO, significantly reducing restore times by eliminating single-threaded bottlenecks in upstream CRIU. The restorer maintains a sliding window of up to 128 concurrent reads in flight via io_submit, keeping the storage device saturated throughout the restore.

The combined effect of AIO and parallel memfd restore is substantial. Here is how the three tested models compare:

ModelCheckpoint sizeUpstream CRIUCRIU (AIO + parallel memfd)Speedup
Qwen3-0.6B6.2 GiB6.8 s2.4 s2.8x
Qwen3-8B26 GiB24 s4.7 s5.1x
gpt-oss-120b129 GiB119 s15 s7.9x

The GPU Memory Service: the real ceiling-breaker

Even with fast CRIU, there is a hard ceiling: model weights dominate the checkpoint size, and cuda-checkpoint cannot restore GPU memory until CRIU has materialized those weights in host memory first. The two steps are serialized, and for a 120B model that is a lot of data to move sequentially.

GMS uses the CUDA Virtual Memory Management (VMM) API to decouple large model weights from the inference worker's process lifetime, offloading the majority of process memory into a separate GMS artifact. By removing weights from the core CRIU checkpoint, GMS allows process state restoration and weight restoration to run concurrently using different memory bandwidth channels. Weight restoration can use the fastest available paths such as GPUDirect Storage (GDS) or peer-GPU RDMA/NVLink.

The impact is dramatic. For gpt-oss-120b, the CRIU checkpoint shrinks from 129 GiB to just 6.7 GiB, with the 74 GiB weight artifact handled separately. When weights are restored over an independent channel, weight restoration can complete in parallel, before CRIU restore even completes. In a proof-of-concept backend that stripes the weights across 8 local NVMe SSDs, the restore process completes in under 5 seconds -- a start-time reduction of 21x for gpt-oss-120b.

What it can and cannot do today

This is an early experimental release. Current limitations include: checkpoint/restore supports vLLM workers only in limited preview; specialized workers (multimodal, embedding, diffusion) are not supported; multi-GPU tensor-parallel configurations have limited validation; GMS restore is not yet available; snapshot-agent must run privileged; and restore is sensitive to live TCP socket state.

The CRIU optimizations (AIO and parallel memfd restore) are also not yet shipped -- these CRIU optimizations are not yet part of Dynamo Snapshot, and will be available once they have been merged into upstream CRIU. The roadmap includes:

  • GMS restore path with pluggable backends (GDS, UCX, etc.), currently gated on a pending CUDA driver patch
  • TensorRT-LLM support
  • Multi-GPU and multi-node support via quiesce/resume hooks for PyTorch, NCCL, and NIXL

Why this matters beyond the headline number

The 21x figure is eye-catching, but the deeper shift is architectural. Today, teams running inference on Kubernetes have to choose between keeping warm replicas on standby (wasting GPU budget) or accepting slow scale-out (risking SLA violations). NVIDIA's Dynamo Snapshot offers a glimpse into what is possible when cutting-edge hardware and software optimizations converge. For enterprises running inference-heavy AI workloads on Kubernetes, this could be a game-changer for cost efficiency, SLA compliance, and user experience.

NVIDIA now has early proof that fast startup for inference workloads on Kubernetes is practical, and is working to stabilize the implementation and expand support to a wider range of workloads. Dynamo Snapshot will roll out incrementally over the coming months. The official documentation covers the current experimental release, which supports single-GPU vLLM and SGLang workloads through the non-GMS checkpoint/restore path.