Skip to main content
Back

Politecnico di Torino Shows Dense Text Breaks LLMs Past 4k Tokens

A new paper shows that how much unique information is packed into a context -- not just its length -- causes LLMs to fail far earlier than expected

Paper, Full Access
  • New variable found: Lexical density -- how much unique information is packed per token -- causes LLM retrieval to fail far earlier than context length alone predicts.
  • Sharp collapse: Models near-perfect on sparse NIAH benchmarks drop below 60% retrieval accuracy on denser benchmarks of identical length and needle position.
  • Early failure threshold: Dense haystacks trigger retrieval failures past just 4k-6k tokens, an order of magnitude earlier than length-based predictions.
  • Three failure modes identified: Conflation (wrong answers), abstention (refusals), and loop inversion -- each tied to specific model families like Llama/DeepSeek, GPT-OSS, and Qwen respectively.
  • Causal, not correlational: Within-benchmark density interventions confirm density directly causes degradation, ruling out task-type as a confound.
  • RAG and agentic implications: Dense retrieved chunks may silently degrade performance; standard NIAH evals overestimate real-world long-context capability. Read the paper.

The long-context race has been dominated by one metric: token count. Models now advertise windows of 128K, 1M, even 10M tokens, and the standard benchmark -- Needle in a Haystack (NIAH) -- confirms near-perfect retrieval scores. But a new paper from researchers at Politecnico di Torino argues that this picture is dangerously incomplete. Dense Contexts Are Hard Contexts introduces lexical density as a third, largely ignored axis of long-context degradation -- and the results are striking.

The problem nobody was measuring

Input length and the position of relevant information are widely cited as the primary causes of degraded LLM long-context performance. But this paper studies lexical density -- the rate at which a context introduces distinct information -- as a third, largely overlooked factor that systematically reduces the effective context window of LLMs.

Think of it this way: a haystack made of repeated boilerplate text ("The UUID for X is Y") is very easy to search, because most tokens are redundant. A haystack made of dense prose -- say, a legal contract or a medical report -- introduces a new concept with nearly every word. Low MATTR indicates a redundant haystack; high MATTR, one where nearly every word is new. The paper's central claim is that models treat these two cases very differently, even when the context length is identical.

This matters because recent developments in LLMs show a trend toward longer context windows, and because these models achieve near-perfect scores on widely adopted benchmarks like NIAH, it's often assumed that their performance is uniform across long-context tasks. However, NIAH is fundamentally a simple retrieval task, in which a known sentence is placed in a long document of unrelated text. Dense real-world inputs -- codebases, contracts, scientific papers, log files -- look nothing like that.

Measuring density with MATTR

The researchers quantify lexical density using MATTR (Moving-Average Type-Token Ratio), a standard linguistics metric. It slides a window across the text and measures what fraction of tokens within that window are unique. A score near 0 means the text is highly repetitive; a score near 1.0 means almost every token is new. The paper models text-intrinsic lexical density via the Moving-Average Type-Token Ratio (MATTR), formalized in the paper.