Skip to main content
Back

NVIDIA's DynoSim Screens Thousands of GPU Configs 1,500x Faster Than Real Time

NVIDIA's DynoSim lets teams simulate thousands of LLM serving configurations at 1,500x real-time speed, cutting GPU validation costs before production deployment.

Repo, Full Access
  • DynoSim is a new Rust-based simulator for NVIDIA's Dynamo inference stack that screens thousands of deployment configs before real GPU validation.
  • It runs at 1,500x real-time speed, simulating a 60-minute serving window in 2.4 seconds on a MacBook Air.
  • Uses discrete-event simulation to model the full stack: scheduler, KV-aware router, Planner autoscaler, and KV cache tiers on one virtual timeline.
  • Key findings: KV-aware routing improves prefix cache reuse by ~18%, host-memory KV tiering cuts TTFT by 19.3%, and the optimal autoscaler interval is 5-10 seconds.
  • Ships free inside the open-source Dynamo repo; entry point is python -m dynamo.replay.
  • Roadmap includes continuous simulation against live production traffic for automatic reconfiguration as workload patterns drift.

Tuning an LLM serving stack is one of those problems that looks manageable until you actually try it. You have to pick a model backend, a tensor-parallel shape (how many GPUs share the model), whether to split prefill and decode across separate workers, how many workers to run, how the router assigns requests, how the KV cache behaves across memory tiers, and when the autoscaler should fire. Every one of those choices interacts with the others, and a tweak that improves throughput can silently blow up latency somewhere else. For large models, even a single realistic experiment can burn through many GPU-hours before you learn whether the idea was worth testing.

NVIDIA's answer to this is DynoSim: a full-stack simulator for the Dynamo inference serving framework that lets you screen thousands of deployment configurations in software before touching real hardware.

Inference engines optimize the GPU itself, while Dynamo adds the distributed layer on top: disaggregated serving, smart routing, KV cache management across memory tiers, and auto-scaling. That distributed layer is where the hard tuning lives, and it is expensive to explore empirically. DynoSim is a workload-driven discrete-event simulation of the Dynamo serving stack that combines measured engine forward-pass timing, scheduler behavior, Router and Planner logic, KV cache effects, and workload traces on one virtual timeline.

The core idea is a simulate-then-verify loop. Instead of running every candidate configuration on a real cluster, teams sweep thousands of options in simulation, shortlist the Pareto-optimal candidates (the configs that offer the best tradeoff between latency and throughput), and only then validate those few on real GPUs. The recommended workflow goes from a single DynoSim run to verify workload shape, then DynoSim sweeps to search the design space, then live Mocker deployments to exercise the real Dynamo frontend without running model inference, and finally real-GPU validation before production rollout.

1,500x faster than real time, on a laptop

The speed claim is the headline number, and it holds up. DynoSim is a full-stack Rust implementation. On an Apple M4 MacBook Air, the single-threaded offline replay simulated the full 23,608-request Mooncake trace with eight round-robin workers in 2.41 seconds of wall time, covering a simulated serving window of 60.1 minutes, about 1,500x faster than real time. That means a sweep over thousands of configurations that would take weeks of GPU time can run in minutes on a developer's laptop.

The speed comes from using discrete-event simulation (DES), a technique where a virtual clock jumps between scheduled events rather than advancing in real time. Components do not wait; they schedule future events with modeled durations (a prefill pass, a KV transfer, a Planner scaling decision) and the runtime jumps to the next timestamp. This is the same class of simulation used in network modeling and chip design.

How the simulation is built

DynoSim is not a single monolithic model. It is a set of composable components that all share the same virtual timeline:

  • Replay harness: Drives request arrivals from a recorded trace or a synthetic workload generator. For multi-turn or agentic traffic, it can wait for completions before issuing follow-up requests.
  • Single-engine simulation (Mocker): Models the scheduler inside one worker, including how requests are batched, how prefill and decode are interleaved, and how KV memory pressure causes preemption. The vLLM and SGLang paths are modeled separately because their schedulers behave differently.
  • AIConfigurator (AIC): Given the model, backend, GPU type, and tensor-parallel shape, AIC estimates how long each prefill or decode pass should take. The Mocker decides what goes into each pass; AIC estimates how long it runs.
  • Multi-engine simulation: Adds the cross-worker behaviors: KV-aware routing, distributed cache state, and Planner autoscaling decisions, all registered as events on the same timeline.
  • Trace collector: Records TTFT (time to first token), TPOT (time per output token), end-to-end latency, throughput, and prefix cache reuse rate.

A key validation result: scheduler-aware replay closes the gap between engine timing estimates and hardware measurements, especially at higher concurrency where TTFT diverges most. Using AIC alone (without the scheduler layer) underestimates TTFT at high concurrency because it does not model request queuing and batching dynamics.

What you can actually discover

The simulation is most useful for three classes of questions that are expensive to answer on real clusters.

Routing policy comparison. In one experiment using the 23,608-request Mooncake toolagent trace on MiniMax-M2.5 FP8 running on NVIDIA HGX B200 hardware, KV-aware routing improved prefix reuse from about 0.38 to 0.44-0.45, reducing TTFT and lifting throughput compared with round-robin placement, though cache-affine placement can increase decode pressure at high concurrency. That tradeoff is exactly the kind of thing you want to know before committing to a routing policy in production.

KV cache tiering. Enabling the G2 host-memory tier (spilling KV blocks from GPU memory to CPU RAM rather than recomputing them) showed a measurable gain: enabling the KVBM G2 host-memory tier reduces prefill recompute by reusing KV blocks that would otherwise be rebuilt, with the largest gain at concurrency 32 where mean TTFT improves by 19.3%.

Autoscaling behavior. This is where DynoSim is arguably most valuable, because autoscaling behavior is inherently macro-scale: it emerges from minutes of traffic, delayed worker startup, and feedback between scale decisions and queues. Three experiments on Qwen3-32B at TP=2 on H200-SXM illustrated this clearly:

  1. Static vs. dynamic deployment: The dynamic deployment with Planner reached a much better cost-latency point: its p90 TTFT and ITL were far lower than any static deployments while using less GPU-hours at the same time.
  2. Scaling interval: P90 TTFT stays about the same from 1-10 second intervals, but scaling events drop sharply from 1,529 to 233. After about 30 seconds, the Planner reacts too slowly to bursts. The best range is around 5-10 seconds.
  3. Cold-start time: For Qwen3-32B at TP=2, the Planner meets the SLA until startup delay reaches about 180 seconds. Around 200 seconds, performance drops sharply, and by 300 seconds the system is stuck behind the traffic burst, with p90 TTFT reaching 242 seconds. This gives teams a concrete engineering target: keep cold-start below 200 seconds or invest in pre-warmed capacity.

The optimizer and the research loop

Beyond one-off experiments, DynoSim includes a sweep optimizer. Today it uses a block-coordinate descent: fix a tensor-parallel shape, search worker counts, then search router settings. It is deliberately simple because the current search space is small enough that coarse search finds good candidates. As the space grows, the same scoring loop can plug into Bayesian search or genetic algorithms.

More ambitiously, the team describes an agentic research loop inspired by Karpathy's autoresearch project: an agent proposes a code change to a router cost function or Planner heuristic, rebuilds Dynamo, reruns the same trace, and keeps only changes that improve the objective. That turns replay into a bounded research loop for router cost functions, Planner heuristics, and cache policies that are awkward to express as a small parameter grid.

Getting started

DynoSim ships as part of the open-source Dynamo repository and is available now at no additional cost. The entry points are straightforward:

  • Single run: python -m dynamo.replay
  • Configuration sweep: dynamo.profiler.utils.replay_optimize
  • Live simulation with Mocker (no GPUs needed): python -m dynamo.mocker

The DynoSim documentation recommends starting with a single run to verify your workload shape and engine arguments, then moving to sweeps once you know the simulation is tracking your real traffic. The key limitation to keep in mind: DynoSim narrows the search space, it does not replace real-hardware validation. The shortlisted candidates still need to be verified on actual GPUs before going to production.

Why this matters beyond NVIDIA's stack

Dynamo is a high-throughput low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments, and it is inference engine agnostic, supporting TRT-LLM, vLLM, SGLang, and others. That means DynoSim is relevant to anyone running multi-GPU inference at scale, not just NVIDIA-first shops.

The deeper shift is conceptual. The field has treated inference deployment as a one-shot decision: profile once at launch, pick a configuration, and revisit it when something breaks. DynoSim points toward a different model: continuous simulation against recorded production traffic, with automatic reconfiguration when the workload drifts. Traffic shape changes over hours and days, and the optimal tensor-parallel shape, prefill/decode split, and router policy from last week may not be optimal today. A simulation loop that runs periodically against live traffic traces could keep deployments tracking the current optimum automatically, without requiring an engineer to notice the drift first.