---
title: "Politecnico di Torino Shows Dense Text Breaks LLMs Past 4k Tokens"
subtitle: "A new paper shows that how much unique information is packed into a context -- not just its length -- causes LLMs to fail far earlier than expected"
url: "https://alphasignal.ai/news/politecnico-di-torino-shows-dense-text-breaks-llms-past-4k-tokens"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-04T00:00:00.000Z"
updated: "2026-06-12T01:52:32.374Z"
type: "Paper"
topics:
  - "Llms"
  - "Benchmarks"
subtopics:
  - "Long Context"
  - "Hallucinations"
upvotes: 1556
source_url: "https://arxiv.org/abs/2606.06203"
comments: 0
truncated: true
---

# Politecnico di Torino Shows Dense Text Breaks LLMs Past 4k Tokens

> A new paper shows that how much unique information is packed into a context -- not just its length -- causes LLMs to fail far earlier than expected

The long-context race has been dominated by one metric: token count. Models now advertise windows of 128K, 1M, even 10M tokens, and the standard benchmark -- Needle in a Haystack (NIAH) -- confirms near-perfect retrieval scores. But a new paper from researchers at Politecnico di Torino argues that this picture is dangerously incomplete. [Dense Contexts Are Hard Contexts](https://arxiv.org/abs/2606.06203) introduces **lexical density** as a third, largely ignored axis of long-context degradation -- and the results are striking.

## The problem nobody was measuring

Input length and the position of relevant information are widely cited as the primary causes of degraded LLM long-context performance. But this paper studies lexical density -- the rate at which a context introduces distinct information -- as a third, largely overlooked factor that systematically reduces the effective context window of LLMs.

Think of it this way: a haystack made of repeated boilerplate text ("The UUID for X is Y") is very easy to search, because most tokens are redundant. A haystack made of dense prose -- say, a legal contract or a medical report -- introduces a new concept with nearly every word. Low MATTR indicates a redundant haystack; high MATTR, one where nearly every word is new. The paper's central claim is that models treat these two cases very differently, even when the context length is identical.

This matters because recent developments in LLMs show a trend toward longer context windows, and because these models achieve near-perfect scores on widely adopted benchmarks like NIAH, it's often assumed that their performance is uniform across long-context tasks. However, NIAH is fundamentally a simple retrieval task, in which a known sentence is placed in a long document of unrelated text. Dense real-world inputs -- codebases, contracts, scientific papers, log files -- look nothing like that.

## Measuring density with MATTR

The researchers quantify lexical density using **MATTR** (Moving-Average Type-Token Ratio), a standard linguistics metric. It slides a window across the text and measures what fraction of tokens within that window are unique. A score near 0 means the text is highly repetitive; a score near 1.0 means almost every token is new. The paper models text-intrinsic lexical density via the Moving-Average Type-Token Ratio (MATTR), formalized in the paper.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMs (arXiv)](https://arxiv.org/abs/2606.06203)
- [Dense Contexts Are Hard Contexts -- Full HTML Paper](https://arxiv.org/html/2606.06203v1)
- [Context Rot: How Increasing Input Tokens Impacts LLM Performance (Chroma Research)](https://research.trychroma.com/context-rot)
- [Long-Context Models vs. RAG: Production Decision Framework](https://tianpan.co/blog/2026-04-09-long-context-vs-rag-production-decision-framework)
- [The Maximum Effective Context Window for Real World Applications](https://www.oajaiml.com/uploads/archivepdf/643561268.pdf)