Perplexity's ROSE Serving Stack Beats vLLM on Speed and Latency

Every Perplexity answer begins with a quiet but expensive step: turning a query and billions of candidate documents into vectors, then finding the closest matches. The company just published a deep dive on the serving stack that powers this, with benchmark numbers showing meaningful wins over vLLM on the same hardware and weights.

The stack runs three services in concert. Ivy is a Rust HTTP gateway handling CPU-side work: JSON parsing, tokenization, input templating, and batch splitting. Tulip is a gRPC inference server built with Rust, tokio, and tonic that owns scheduling and batching. ROSE (Runtime-Optimized Serving Engine) runs the model forward passes in Python and manages CUDA graphs.

Two workloads, one kernel path

Search embeddings arrive in two very different shapes. Batch embedding runs during database construction or re-indexing and needs maximum throughput to minimize cost. Online embedding runs on short live queries and needs minimum latency. Perplexity maps these onto LLM inference primitives: batch embeddings behave like compute-bound prefill, and online embeddings on a handful of tokens behave like memory-bound decode. The team reuses its optimized prefill and decode kernels for both. Inside ROSE, the main adaptation is skipping the KV cache and dispatching to attention kernels that support ragged inputs, which avoids padding overhead.

When the CPU becomes the bottleneck

Embedding models are small, which flips the usual assumption that the GPU is the slow part. On small batches, CPU-side work like scheduling and kernel launching can outweigh the GPU computation itself. Tulip addresses this with two techniques.

• CUDA graphs. A CUDA graph captures the metadata needed to launch all the kernels of a forward pass with a single driver call, eliminating repeated Python and PyTorch overhead. Because embedding shapes vary, graphs are captured per sequence-count and token-count combination, with token counts padded to buckets of 64 or 256. This can produce thousands of graphs requiring minutes to capture, so Tulip captures them lazily as live traffic arrives.
• LazyTensors. A LazyTensor tracks a host buffer in page-locked memory alongside a cudaMemcpyAsync event, letting an async Rust task block on the previous batch's result while the CPU prepares the next one.

Together, these keep the CPU one batch ahead of the GPU rather than stalling on kernel launches.

A scheduler built around token budgets

Tulip's scheduler is deliberately simple. For small embedding models at typical serving lengths, the linear cost of dense layers dominates the quadratic cost of attention, so latency tracks token count rather than sequence count. Once a batch hits roughly 512 tokens on a sub-billion-parameter model, the GPU is saturated and adding more sequences yields nothing. Tulip therefore pulls sequences first-come, first-served until it hits that token budget.

Keeping multiple attention backends

ROSE supports several attention backends simultaneously. The team integrated FlashInfer 2, FlashInfer 3, and FlashAttention 4 kernels for ragged attention. FlashAttention 4 is generally faster, but FlashInfer 3 outperforms it on Qwen-based models at very long sequence lengths. The optimal choice depends on head count and dimension, so ROSE keeps all of them available.

Benchmark results

Perplexity benchmarks against vLLM v0.22.0 in BF16 on real model weights and inputs, with warmup runs verifying cosine similarity within 0.1%. Across four scenarios (low-latency single-query embedding, batch scoring, high-throughput indexing, and concurrent requests), Tulip beats the baseline on both p50 and p99 for BGE-M3 and pplx-embed-1-0.6b at 128, 512, and 4096 token lengths.

Takeaways for teams not running at Perplexity's scale

Most teams will keep using vLLM, TEI, or a hosted embeddings API, and that is reasonable. The writeup is still a useful blueprint for anyone hitting the ceiling of off-the-shelf serving:

• If embedding traffic skews toward short queries, kernel launch overhead is probably consuming your latency budget. CUDA graphs combined with lazy result tracking are the fix.
• Sharing kernels between an LLM stack and an embedding stack is viable, provided you swap paged attention for ragged attention and drop the KV cache.
• Rust in the hot path for tokenization, HTTP, and gRPC is increasingly common. vLLM, SGLang, and TokenSpeed are all pulling Rust or C++ into their stacks for the same reasons.

Attention kernels for embeddings are largely a solved problem. The performance gains now live in the plumbing around them: how quickly you can feed the GPU, how much CPU work you can hide, and how tightly your scheduler matches the actual shape of incoming traffic.