Novita AI's PegaFlow Stops vLLM From Losing Its Entire Cache on Restart

PegaFlow moves vLLM's KV cache into a standalone Rust daemon, surviving restarts and delivering up to 72% higher throughput by sharing cache across instances and nodes via RDMA.

·
·
·
Novita AI's PegaFlow Stops vLLM From Losing Its Entire Cache on Restart
  • PegaFlow is an open-source external KV cache daemon for vLLM, built by Novita AI, that survives engine restarts and shares cache across instances and nodes.
  • Benchmarks show 56% higher throughput for 8x Qwen3-8B instances sharing one pool vs. 8 isolated caches, and 72% higher throughput for DeepSeek-V3.2 MLA TP8.
  • Startup time drops 2.15x (71.4s to 33.2s) when the 500 GiB KV pool is pre-owned by the external daemon rather than vLLM.
  • Built in Rust with a GIL-free data path; uses CUDA IPC + gRPC to connect to vLLM workers with no vLLM source changes needed.
  • Three-level cache hierarchy: pinned DRAM (L1), remote DRAM via RDMA at 194 GB/s average (L2), and local SSD via io_uring at ~6.5 GB/s (L3).
  • Free and open source (Apache 2.0) at novitalabs/pegaflow; requires vllm>=0.20.0 and plugs in via kv_transfer_config.

Every time vLLM restarts, hundreds of gigabytes of carefully warmed KV cache vanish. For teams running production inference fleets, that means slower cold starts, wasted GPU cycles re-computing prefills, and cache hit rates that reset to zero after every upgrade or crash. PegaFlow, built by Novita AI in collaboration with the vLLM team, is a direct fix: a standalone external KV cache service that outlives the inference engine itself.

PegaFlow integrates with vLLM as an external KV cache service, implemented as a standalone Rust process and connected through the external KV connector interface. It moves KV cache lifetime out of the vLLM worker process, pools cache across local instances and remote nodes, and combines pinned host memory, RDMA-accessible remote memory, and SSD into a three-level cache hierarchy.

The problem nobody talks about enough

KV cache (the stored attention keys and values for previously processed tokens) is one of the most expensive assets in LLM serving. It takes time to allocate, time to warm, and can occupy hundreds of gigabytes per host. The problem is that in every conventional vLLM deployment, that cache lives inside the inference process itself.

In a conventional in-process design, that asset is tightly coupled to the inference engine process. This coupling becomes painful during engine crashes, rolling upgrades, and model switches. When an engine restarts, the host KV pool disappears with it. For a production fleet doing rolling deploys, this is a constant tax on latency and throughput.

There is a second, less obvious problem: isolation. When multiple small-model instances run on one host, running eight Qwen3-8B instances on an 8-GPU host can store the same system prompt eight times. For models such as DeepSeek-V3.2, the logical latent KV can be stored once, but an in-process TP8 deployment may physically store it once per rank. The cache budget is the same, but most of it is wasted on duplicates.

A daemon that owns the cache

PegaFlow addresses this by moving the KV cache runtime into a standalone daemon on each machine. The PegaFlow server owns the host KV pool, SSD cache, topology metadata, RDMA resources, indexing state, and background tasks. vLLM workers connect to the local PegaFlow process through CUDA IPC on the data path and gRPC on the local control path.

PegaFlow architecture: vLLM/SGLang process connects to PegaEngine which manages GPU KV Cache, host memory, SSD, and cross-node RDMA via MetaServer

The failure domain separation is the key insight. A vLLM process can crash, upgrade, or switch models while the cache service remains alive. Conversely, cache-layer issues do not have to bring down the inference engine process. Different models, tensor-parallel configurations, and engine versions can share one PegaFlow process with namespace isolation.

The choice of Rust for the data plane is not just a performance flex. PegaFlow's data plane avoids Python interpreter overhead, GIL contention, and stop-the-world garbage collection. This matters because a production cache service does more than move bytes on the critical path. It also runs background tasks such as statistics collection, index uploads, prefetching, health checks, metrics reporting, eviction, and SSD cache management. In Python, those background tasks compete with the serving path for the GIL. In Rust, they don't.

What the numbers look like

The benchmarks are structured around a fixed cache budget, testing how much more useful that budget becomes when it is shared rather than siloed. The results are significant:

  • 2.15x faster startup when a 500 GiB host KV pool was already owned by the external cache service.
  • 56% higher throughput for eight Qwen3-8B instances sharing one host cache instead of eight isolated caches.
  • 72% higher throughput for DeepSeek-V3.2 MLA with TP8 by storing logical KV once instead of once per TP rank.
  • 194 GB/s average remote-read throughput for large prefix pulls in an internal RDMA cluster with 8 x 400 Gbps NICs per node.

A public benchmark on H800 with Llama-3.1-8B, using 8 prompts, 10K-token prefill, 1-token decode, and 4.0 req/s shows the warm cache path reduces mean TTFT from 572.5 ms to 61.5 ms, with P99 TTFT dropping from 1113.7 ms to 77.0 ms. That is roughly a 9x improvement in time-to-first-token on cache hits.

The MLA (Multi-head Latent Attention, DeepSeek's compressed attention mechanism) result deserves special attention. In a standard TP8 deployment, each of the 8 tensor-parallel ranks stores its own copy of the KV cache. PegaFlow stores the logical KV once and serves all ranks from it, effectively multiplying the usable cache capacity by 8x for that layer.

Three levels of storage, one coherent pool

PegaFlow uses a three-level cache hierarchy. Hot local blocks stay in pinned DRAM, remote hits can be fetched over RDMA, and colder reusable blocks can spill to local SSD. Here is how the levels break down:

LevelMediumAccess pathRole
L1Local pinned DRAMCUDA IPCFast local KV reuse
L2Remote DRAMRDMA READCross-node cache sharing
L3Local SSDio_uringLarge-capacity spillover

The SSD cache is implemented in Rust on top of io_uring. In internal tests, a single SSD delivered roughly 6.9 GB/s peak read throughput. PegaFlow keeps online steady-state throughput around 6.5-6.6 GB/s per disk, trading about 5% peak bandwidth for more stable tail latency. With RAID0 across multiple disks, total throughput scales approximately linearly.

For cross-node transfers, RDMA (Remote Direct Memory Access) lets one machine read another's memory without involving the remote CPU. At the measured average throughput, a 24 GiB KV cache segment can be pulled from a remote node in roughly 100 ms. That can replace a prefill computation that would otherwise consume seconds of GPU time.

P99.9 tail latency comparison: Rust Tokio shows dramatically more stable latency than Python uvloop and ZMQ under GIL load at concurrency levels 16 and 64

Smart eviction and knowing your ceiling

Cache eviction policy matters more than it might seem. Simple LRU (Least Recently Used) can be wrecked by scan-like traffic where many one-time blocks pass through and flush out reusable content. PegaFlow can enable a TinyLFU admission policy. This policy admits blocks only when they are likely to be reused, protecting the cache from one-time traffic. TinyLFU is disabled by default because the best admission policy depends on workload shape.

There is also a genuinely useful observability feature baked in: a theoretical hit-rate ceiling estimator. PegaFlow estimates the theoretical hit-rate upper bound online using HyperLogLog. By placing measured hit rate and theoretical upper bound on the same dashboard, operators can distinguish whether the cache is already close to the workload ceiling, whether the measured hit rate is far below the ceiling suggesting room for better capacity or prefetching, or whether the theoretical ceiling itself is low indicating the workload has limited reuse. This turns cache tuning from guesswork into a measurable engineering problem.

Cache policy comparison (FIFO, LRU, TinyLFU) across different cache sizes showing TinyLFU outperforming recency-based policies under small cache and scan-heavy conditions

Zero vLLM source changes required

This integration is exposed through the existing kv_transfer_config path. PegaFlow can be used as an external cache backend without modifying vLLM source code or carrying a long-lived fork. That is a meaningful constraint: it means PegaFlow tracks upstream vLLM rather than diverging from it.

Getting started requires vllm>=0.20.0. Install the package and start the server:

css
uv pip install pegaflow-llm  # CUDA 12
pegaflow-server \
  --pool-size 30gb \
  --ssd-cache-path /path/to/ssd-cache \
  --ssd-cache-capacity 512gb

Then connect vLLM to it:

haskell
vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "PegaKVConnector",
    "kv_role": "kv_both",
    "kv_connector_module_path": "pegaflow.connector"
  }'

PegaFlow also works with SGLang, not just vLLM. The project is open source under Apache 2.0 and available at novitalabs/pegaflow. There is no cost to use it beyond the infrastructure you are already running.

Who actually benefits from this

PegaFlow is most valuable in specific production patterns:

  • Multi-instance small model serving -- running many copies of a model (e.g., 8x Qwen3-8B) on a single host, where system prompts and common prefixes get recomputed per-instance today.
  • Large TP deployments with MLA models -- DeepSeek-V3.2 and similar architectures where per-rank KV storage wastes most of the cache budget.
  • Frequent rolling upgrades -- any fleet where restarts are common and cold-start latency is a real cost.
  • Multi-node inference clusters with RDMA -- where a cache hit on one node can serve a request routed to another, avoiding expensive prefill recomputation.
  • Agentic workloads -- long conversations with repeated system prompts that benefit heavily from prefix caching across turns and sessions.

The broader implication is a shift in how the field thinks about KV cache. Treating it as ephemeral process state made sense when models were small and restarts were rare. At production scale with 500 GiB pools, multi-model fleets, and rolling deploys, the cache is infrastructure -- and it deserves to be managed like infrastructure.

Trending
  • No trending articles

Comments

avatar

Next Reads