Skip to main content
Back

Perplexity Rebuilds Its Tokenizer in Rust, Cuts CPU Load by 5x

Perplexity open-sources a Rust Unigram tokenizer that cuts CPU latency 5x over HuggingFace, reaching 63µs at 514 tokens with zero heap allocations

Repo, Full Access
  • Perplexity open-sourced a Rust Unigram tokenizer that cuts CPU utilization 5-6x in production reranker pipelines.
  • At 514 tokens, it runs in 63µs with zero heap allocations, vs 349µs for HuggingFace tokenizers.
  • Key innovations: double-array trie, per-node bitmap packing into 64-byte cache lines, and Linux huge pages for the trie.
  • Targets XLM-RoBERTa's 250K-token vocabulary, widely used for ranking, retrieval, and similarity tasks.
  • The root cause was HuggingFace's tokenizer allocating a fresh string and doing a hash-map lookup per trie match (7,295 allocs at 512 tokens).
  • Code is available at pplx-garden under MIT license, with a full technical writeup.

Tokenization is the unglamorous plumbing of every inference pipeline, and for most teams it is an afterthought. Perplexity's new research post makes a compelling case that it shouldn't be. The company rebuilt its Unigram tokenizer from scratch in Rust, cut CPU utilization by 5-6x in production, and is now open-sourcing the implementation for everyone to use.

At production input lengths, the new encoder cuts p50 latency by roughly 5x versus the Hugging Face tokenizers crate, ~2x versus SentencePiece (C++), and ~1.5x versus IREE's tokenizer (C), with zero steady-state heap allocations. The code is live in the pplx-garden repository under an MIT license, available right now.

When the GPU waits on the CPU

LLM inference cost is usually framed as GPU work: KV caches, attention kernels, expert routing. But a meaningful portion of production traffic runs on much smaller models, two to three orders of magnitude smaller than frontier transformers. Embedding models, classifiers, and rerankers still run on GPUs, but the forward pass is so short that everything around it starts to matter.

The clearest case is a reranker scoring hundreds of candidate documents per request. With a small model, GPU compute typically finishes in single-digit milliseconds, but every input still has to go through CPU-side tokenization first, turning text into vocabulary IDs. When you're processing a large batch, that CPU step can eat a meaningful fraction of your total request latency, and no amount of GPU optimization will fix it.

This work targets XLM-RoBERTa, a 250K-token Unigram vocabulary trained with SentencePiece. Fine-tuned RoBERTa-family encoders are a common choice for ranking, retrieval, and similarity tasks.

What was broken about the original

Unigram tokenization (introduced by Kudo in 2018) frames text segmentation as a shortest-path problem: each vocabulary token has a learned log-probability, and the encoder picks the segmentation whose scores sum highest. This is solved with the Viterbi algorithm, a classic dynamic programming technique that walks through byte positions and finds the optimal sequence of tokens.

Hugging Face's tokenizers v0.22 implements the recurrence directly, with a hash-map for the reverse lookup and hash-map-children at every trie node. The problem is what happens inside the inner loop: every prefix-search match allocates a fresh string and then probes the hash-map.

The three main bottlenecks Perplexity identified:

  • Allocation per match: 7,295 allocations at 512 tokens, 14,542 at 1k, 299,171 at 16k.
  • Pointer chasing per byte: a HashMap at every trie node, hashing the next byte and dereferencing two pointers per step.
  • L2 cache thrashing on long inputs: L2 miss rate climbs from 12% at 128 tokens to 50% at 16k as the cumulative allocation footprint exceeds the per-core 2 MB L2.
Bar chart showing reference heap allocations in MB per encode, growing exponentially from 0.04 MB at 16 tokens to 39 MB at 16k tokens

Three optimizations, stacked

The team didn't just patch the existing code. They rebuilt from the ground up with three compounding changes:

  1. Double-array trie: SentencePiece and IREE both use the double-array trie introduced by Aoe in 1989. The trie is encoded in two flat integer arrays, base and check. A child lookup is just two array reads, an integer add, and a comparison, with no hashing, no pointer chasing, and no per-node heap allocation. Switching to this structure alone dropped p50 latency from 155µs (baseline) to 68µs at 514 tokens.

  2. Bitmap + inline packing: The Darts trie needs two dependent loads per byte step. The team replaced the check array with a per-node bitmap that records which bytes have children. The bitmap lookup compiles to a single bit test against one 64-bit word, which runs in roughly a cycle. They also packed all four per-node fields (bitmap, base, token ID, and score) into a single 64-byte cache line, so one memory load fetches everything needed for a node visit. The trade-off: the trie grows from ~9 MB to ~50 MB.

  3. Huge pages: The trie data is roughly 50 MB. On a default Linux process, that memory is backed by 4 KB pages, so the full trie spans about 12,000 distinct virtual pages, far past what the CPU's first-level data TLB (Translation Lookaside Buffer) can hold. Each Viterbi step touches a different node, and over a 512-token encode this burns roughly 9,000 cycles in page-table walks. Backing the trie with 2 MB huge pages instead collapses those 12,000 pages to just 25, eliminating the TLB pressure entirely.

Heatmap comparison table showing per-encode latency in microseconds across input sizes for all tokenizer implementations at P50 and P99

The final numbers

In production this reduced CPU utilization in Perplexity's inference stack by 5-6x and shaved double-digit milliseconds off reranker latency. The per-encode instruction count dropped from 3.66M to 1.04M, a 3.5x reduction. At 514 tokens, the encoder runs in 63µs with zero heap allocations on every call.

Here's how it stacks up against the competition at 514 tokens:

Implementationp50 LatencyInstructionsAllocations
Hugging Face349 µs3.60M7,295
SentencePiece (C++)128 µs1.83M1,559
IREE (C)112 µs2.28M1
pplx-garden63 µs1.04M0

It's worth noting that the off-the-shelf Rust wrapper crates around SentencePiece and IREE illustrate the same point in reverse: on top of native C and C++ tokenizers, the FFI boundary and per-call allocations the wrappers introduce are visible in the latency numbers. The sentencepiece Rust crate measures 1.6-1.9x higher latency than the native binary at every input size. This is a useful reminder that wrapping a fast library in a slow interface doesn't make the interface fast.

What this means for your stack

This is most relevant if you're running rerankers, embedding models, or classifiers at scale, especially any XLM-RoBERTa-based model. In production, Perplexity parallelizes across cores using rayon , Rust's data-parallelism library, so the single-threaded numbers above can be further scaled horizontally.

The deeper lesson here is about where to look for latency wins. The bulk of the latency reduction came from lightweight changes that removed wasted work: dropping per-encode allocations, replacing the HashMap-children trie with a double-array, and stripping the generic-API overhead. Using the existing per-node bitmap as the validity check and backing the trie with 2 MB huge pages combined accounted for 10-20% on top of the larger switch to a double-array trie. None of these required new algorithms or hardware, just careful attention to what the CPU is actually doing.

The implementation is available now at github.com/perplexityai/pplx-garden, free to use under MIT. If you're building retrieval pipelines or reranking systems, it's worth a look before your next GPU upgrade.