Skip to main content
Back

Perplexity Cuts RAG Context Tokens by 70% Without Sacrificing Accuracy

Perplexity's new query-aware compression cuts context tokens by up to 70% while boosting answer accuracy — now live in its apps and API.

  • Live now: Perplexity deployed query-aware context compression across its apps and API Platform, cutting context tokens by up to 70%.
  • Better accuracy: Vital content per snippet is up 63%; BrowseComp accuracy improved by up to 4.81 percentage points over baseline.
  • 50x compression ratio: On SimpleQA, the medium preset hits 95% accuracy at ~200 tokens per document from 10K+ raw tokens.
  • Fast enough for production: The distilled 17-layer model runs at under 20ms (p99), with 35–40% lower inference latency vs. the 28-layer teacher.
  • Trained on 750K examples: An LLM-as-a-judge pipeline labeled query-document pairs at span level; binary cross-entropy trains the compression head.
  • Extractive, not generative: Source text is preserved verbatim for citation fidelity — no paraphrasing or hallucination risk from the compressor.

Every RAG system faces the same dirty secret: most of what gets retrieved is junk. Nav bars, cookie banners, ads, boilerplate footers , all of it gets shoved into the context window alongside the one sentence that actually answers the question. Perplexity just shipped a production fix for this, and the numbers are striking.

The company deployed a new, state-of-the-art snippet generator within its applications and API Platform. At its core is a query-aware context compression model that decides, for each query and candidate result, the subset of spans that should be preserved. The result: context tokens cut by up to 70% while improving answer quality , a combination that most teams assume is a tradeoff, not a win-win.

The "Context Rot" Problem

Long streams of irrelevant information waste model capacity, resulting in "context rot" that impairs a model's ability to address the actual user request. This isn't just an accuracy problem. Passing distractors to a downstream agent creates three compounding problems: it hurts accuracy, regresses latency, and raises cost. Every unnecessary token you feed the model makes it slower and more expensive , and you get worse answers for the privilege.

The standard industry response has been to throw more context at the problem. Bigger context windows, more retrieved chunks, higher token budgets. Perplexity's bet is the opposite: be more surgical about what you pass downstream in the first place.

Extractive, Not Generative

The key design decision here is what kind of compression to use. There are two broad approaches: generation-based (ask a model to write a summary of the page) and selection-based (pick the relevant spans and drop the rest). Perplexity ruled out generation, since creating a new summary for every page hurts performance across accuracy, latency, and cost. Summarization can paraphrase the source, making citation alignment harder, and can introduce wording not present in the page , risky when the snippet is used as evidence.

Instead, they define the task as query-aware extractive compression: separating the helpful parts of the retrieved context from the distracting parts. The model accepts a query and candidate result, decides which spans to preserve, and keeps source text rather than rewriting it. This makes the result easier to cite, easier to verify, and cheaper to pass downstream.

How the Model Was Built

The backbone is a pplx-diffusion model , the same model family behind pplx-embed-v1. A key property of this family is bidirectional context: each token representation can attend to the full query and the full candidate context before the model decides what should be kept. This is different from a standard left-to-right decoder, which can only look backwards.

On top of this backbone, they train a lightweight compression head that predicts which parts of the context should be kept. Because no generation is involved, these predictions are made in parallel across the entire context. At serving time, the snippet engine aggregates predictions at the sentence level, applies a threshold, and trims the result to the requested token budget.

Training required granular supervision , document-level relevance labels are too coarse for this task. Here's how they built the training data:

  • An LLM-as-a-judge pipeline first performs query understanding, analyzing the query and producing a list of possible user intents.
  • The judge then processes the candidate context and returns spans copied verbatim from the text, with a category for each span.
  • Naive string matching recovered 98% of spans; adding a simple regex-based pass brought span recovery to 100%.
  • This pipeline labeled 750,000 query-document pairs, used to train the compression head with binary cross-entropy.

Shrinking the Model for Production

A compressor that runs before every LLM call has to be fast , otherwise the latency savings from shorter context are eaten by the compression step itself. Perplexity compared three model depths: 28 layers (full teacher), 17 layers (pruned), and 11 layers.

Line graph comparing compression performance across 28-layer, 17-layer pruned, 17-layer distilled, and 11-layer model variants

The 11-layer model demonstrated the risks of compressing too aggressively. The pruned 17-layer model closely tracks the 28-layer model in the saturated regime, but a small gap opens up in the steep portion of the curve where most production traffic operates.

To close that gap, they used knowledge distillation , augmenting the standard loss with a soft cross-entropy term on the teacher model's predictions. This brought the student model to parity, allowing a full switch to the distilled 17-layer model in production. The result: a 35–40% decrease in inference latency and 40–45% decrease in aggregate GPU compute with no quality drop relative to the 28-layer teacher. Production latency clocks in at under 20 ms (p99). That's fast enough to be invisible in the serving path.

The Benchmark Results

Perplexity tested compression across two benchmarks using their Agent API with three context presets (low: 1K tokens, medium: 4K, high: 10K per result set).

BrowseComp (multi-step agentic research): Compression delivers a new Pareto frontier across the board. For each setting, introducing compression reduces query-level token usage by anywhere from 10% to 70%, with accuracy gains between 4 and 4.81 percentage points. Introducing compression allows even the "low" preset to outperform the no-compression "high" preset, while reducing token usage by more than three-fold.

BrowseComp accuracy vs. average total tokens per task, showing Perplexity with compression outperforming baselines and competitors

SimpleQA (single-step factual lookup): With compression enabled, the medium preset achieves 95% accuracy with an average of only 200 tokens per document. Given the average document length of over 10,000 tokens, this represents frontier-level performance at a compression ratio of more than 50.

There's also a subtle non-obvious effect worth noting: with compression, even increasing per-step context size may enable the model to take fewer steps to arrive at the correct result , meaning the token savings compound across a multi-step agent loop, not just within a single retrieval call.

What Actually Gets Cut

To understand what the compressor is doing at the token level, Perplexity ran a separate evaluation classifying every token in every retrieved snippet into six categories: vital, off-topic, ads, metadata, UI/navigation, and duplicate.

Compression increases the proportion of vital tokens per snippet by an average of 63% over the no-compression baseline. There's a corresponding decrease of 29% in irrelevant tokens, with particularly steep drops in UI/navigation (-58%), metadata (-46%), and ads (-43%).

The categories being cut are exactly what you'd expect from a raw web scrape: the stuff that's useful for a browser but noise for a language model.

What This Means for API Users

This compression is already live in Perplexity's consumer product and across its API Platform. If you're building on the Agent API, you get this automatically. The approach creates a simple control surface, allowing systems and API customers to tune the degree of compression to balance between recall, latency, and cost.

The practical implications for anyone building RAG pipelines or search-grounded agents:

  • Lower token costs , fewer tokens passed to the answer model means lower per-query spend
  • Better accuracy , less noise means the model can focus on what matters
  • Faster responses , shorter context = faster prefill and fewer reasoning tokens
  • Smaller context budgets work harder , the "low" preset now beats the uncompressed "high" preset on BrowseComp

The Bigger Shift

The dominant assumption in RAG has been that more context is safer , if you're not sure what's relevant, include everything and let the LLM figure it out. This work is a direct challenge to that assumption. A good snippet should promote accuracy by presenting the precise evidence needed to ground the model's response, reduce latency and cost by removing irrelevant context before the LLM call, and ensure traceability by preserving citation fidelity and keeping source wording intact.

The fact that a 17-layer distilled encoder running in under 20ms can reliably outperform throwing raw web pages at a frontier model suggests the field has been underinvesting in the retrieval-to-context layer. The answer isn't always a bigger model or a longer context window , sometimes it's a smarter filter.