---
title: "NetKV Cuts AI Response Latency 17.6% by Making Schedulers Network-Aware"
subtitle: "NetKV adds a network-cost oracle to LLM inference schedulers, cutting mean TTFT by up to 21% on 64-GPU clusters with zero changes to hardware or inference engines"
url: "https://alphasignal.ai/news/netkv-cuts-ai-response-latency-17-6-by-making-schedulers-network-aware"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-02T00:00:00.000Z"
updated: "2026-06-10T16:11:34.066Z"
type: "Paper"
topics:
  - "Infra"
  - "Training Infra"
subtopics:
  - "Inference Optimization"
  - "Model Serving"
  - "Distributed Training"
upvotes: 1020
source_url: "https://arxiv.org/abs/2606.03910"
comments: 0
truncated: true
---

# NetKV Cuts AI Response Latency 17.6% by Making Schedulers Network-Aware

> NetKV adds a network-cost oracle to LLM inference schedulers, cutting mean TTFT by up to 21% on 64-GPU clusters with zero changes to hardware or inference engines

Every time a large language model finishes processing your prompt, it has to ship the result -- a dense blob of floating-point data called the KV cache -- across a datacenter network before it can start generating your response. In modern **disaggregated inference** setups, where prefill (prompt processing) and decode (token generation) run on separate GPU pools, that network hop is unavoidable. And yet, today's schedulers route requests as if the network doesn't exist. A new paper, [NetKV](https://arxiv.org/abs/2606.03910), fixes that blind spot.

## Why disaggregated inference is the new normal

Prefill-decode (PD) disaggregation separates the prefill phase from the decode phase, assigning them to dedicated prefill engines and decode engines respectively. The two phases exhibit distinct compute and memory patterns: prefill is compute-intensive and batched, while decode is memory-bound and latency-sensitive. Keeping them on the same GPU means one always starves the other.

Instead of forcing prefill and decode to share the same GPU, disaggregated systems split them into different sets of GPUs, which fundamentally eliminates interference between prefill and decode, and allows each to be independently optimized for throughput or latency. Systems like [Mooncake](https://arxiv.org/html/2407.00079v1) \-- the serving platform behind Kimi -- have taken this architecture to production scale. Mooncake is currently operational across thousands of nodes, processing over 100 billion tokens daily.

But disaggregation introduces a new bottleneck: after completing the prefill computation for a request, the prefill instance transfers both the request and its corresponding KV cache to a decode instance via high-speed interconnects such as NVLink or InfiniBand. That transfer time lands directly inside the Time to First Token (TTFT) budget -- the latency the user actually feels.

## The gap current schedulers leave open

Today's state-of-the-art schedulers make two considerations when routing a request to a decode instance: compute load and prefix-cache locality (whether the decode GPU already holds a cached version of a matching prefix). Both are sensible. Neither accounts for where the decode instance sits in the network relative to the prefill instance that just finished.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference](https://arxiv.org/abs/2606.03910)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://dl.acm.org/doi/10.1145/3773772)
- [Introducing Disaggregated Inference on AWS powered by llm-d](https://aws.amazon.com/blogs/machine-learning/introducing-disaggregated-inference-on-aws-powered-by-llm-d/)
- [Disaggregated Inference: 18 Months Later | Hao AI Lab](https://haoailab.com/blogs/distserve-retro/)
- [Serving Agentic Workloads at Scale with vLLM x Mooncake](https://vllm.ai/blog/2026-05-06-mooncake-store)