Georgetown Proves Central Story Entities Make AI Language Models More Predictable
A new study shows that important entities in text are locally harder to predict but make surrounding content more predictable, refining the Uniform Information Density theory.
ProTakeaways
- Georgetown study of 70K entity mentions across 16 genres refines the Uniform Information Density hypothesis with a new discourse pressure.
- Globally salient entities are locally more surprising but reduce surprisal for surrounding content when used as prompts.
- A novel minimal-pair prompting method controls for position, length, and nesting confounds via DistilGPT-2 surprisal.
- Effect is strongest in academic writing, biographies, and fiction; weakest in conversation, Reddit, and vlogs.
- Salience-conditioned sentence surprisal drops up to 0.47 standard deviations relative to non-salient entity prompts.
- Built on the GUM-SAGE dataset, with results replicated on GPT-2 Small.
A long-standing idea in computational linguistics called Uniform Information Density (UID) says that speakers and writers try to spread information evenly across a text, avoiding sudden spikes or lulls in how surprising each word is. Recent research has poked holes in this picture by identifying syntactic and phonological pressures that create systematic bumps in the surprisal contour. A new paper from Georgetown University adds a discourse-level pressure to that list: the global salience of the entities a text is fundamentally about.
Using 70,000 manually annotated entity mentions across 16 English genres, Jessica Lin and Amir Zeldes show that globally salient entities are locally more surprising than throwaway ones, yet they systematically reduce the surprisal of the surrounding document when used as prompts. The effect is strongest in topic-coherent texts like academic writing and fiction, and nearly disappears in Reddit threads and vlogs.
Where the old UID picture fell short
Surprisal is the negative log-probability a language model assigns to a token given its context. UID predicts that this quantity stays roughly flat across a well-formed discourse. Prior work has identified syntactic constraints, phonological factors, and discourse structures as pressures that create departures from uniformity, but no one had asked whether the discourse-level importance of entities plays a role.