---
title: "Florida Atlantic University Finds GPT-2's Hidden States Predict Human Reading Better Than Surprisal"
subtitle: "A new metric derived from transformer hidden states predicts human reading times independently of surprisal, suggesting LLMs and humans share deeper processing dynamics than previously known"
url: "https://alphasignal.ai/news/florida-atlantic-university-finds-gpt-2-s-hidden-states-predict-human-reading"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-03T00:00:00.000Z"
updated: "2026-06-15T01:02:38.568Z"
type: "Paper"
topics:
  - "Llms"
  - "Reasoning"
subtopics:
  - "Hallucinations"
  - "Test Time Compute"
upvotes: 357
source_url: "https://arxiv.org/abs/2606.05346"
comments: 0
truncated: true
---

# Florida Atlantic University Finds GPT-2's Hidden States Predict Human Reading Better Than Surprisal

> A new metric derived from transformer hidden states predicts human reading times independently of surprisal, suggesting LLMs and humans share deeper processing dynamics than previously known

One of the most striking convergences in modern AI research is that language models seem to predict human reading behavior remarkably well. Feed a sentence through GPT-2, compute how surprised the model is at each word, and those surprise values correlate with how long humans take to read each word. This has been taken as evidence that humans and LLMs are doing something similar: both appear to run a version of next-token prediction. But a new [paper from Florida Atlantic University](https://arxiv.org/abs/2606.05346) argues that surprisal only tells half the story, and that looking inside the model reveals something even more interesting.

## The Problem with Surprisal

Surprisal is defined as the negative log probability of a word given its context. It is a single number, computed once per word, that collapses everything the model knows about the preceding sentence into one scalar. Surprisal reduces rich sequential representations to a single scalar at each word, discarding information about the direction in which the interpretation has been evolving. Think of it like tracking someone walking through a city: surprisal tells you whether they ended up somewhere unexpected, but says nothing about the direction they were heading before they got there.

There is also a known paradox with surprisal: larger transformer-based pre-trained language models with more parameters and lower perplexity nonetheless yield surprisal estimates that are less predictive of human reading times. In other words, as models get smarter, their surprisal values diverge from human behavior. This is a red flag that surprisal is not capturing the full picture of what humans are doing.

## A New Measure: Trajectory Extrapolation Error

The paper introduces a measure called **trajectory extrapolation error**. The idea is geometrically intuitive: as a transformer processes a sentence word by word, its hidden states (high-dimensional vectors representing the model's internal understanding) trace a path through representational space. At each new word, you can ask: given the direction the hidden states were moving over the last few words, where would we _expect_ the next hidden state to land? And how far did it actually land from that predicted spot?

Here is the core computation:

```
import numpy as np
def trajectory_extrapolation_error(hidden_states, k=3):
    """
    hidden_states: list of vectors h[t-k], ..., h[t-1], h[t]
    Returns: Euclidean distance between predicted and actual h[t]
    """
    window = np.array(hidden_states[-(k+1):-1])  # k preceding states
    t_vals = np.arange(k).reshape(-1, 1)
    # Fit linear trajectory via OLS
    coeffs = np.linalg.lstsq(np.hstack([t_vals, np.ones_like(t_vals)]),
                              window, rcond=None)[0]
    predicted = coeffs[0] * k + coeffs[1]        # extrapolate one step
    actual = hidden_states[-1]
    return np.linalg.norm(actual - predicted)     # Euclidean distance

```

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal](https://arxiv.org/abs/2606.05346)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00548/115371/Why-Does-Surprisal-From-Larger-Transformer-Based)
- [Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens](https://arxiv.org/abs/2304.11389)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://arxiv.org/pdf/1903.03260)