Harvard and IBM Find LLMs Hide Secret Self-Knowledge Only for Factual Recall

Do large language models actually know things about themselves that outsiders can't figure out? A team from the Technion, IBM Research, and Harvard's Kempner Institute took a hard look at this question in a new paper, and their answer is more nuanced than either side of the ongoing debate had suggested. LLMs carry privileged self-knowledge about correctness, but only for factual recall. On math problems, their internal states expose no secret sauce that a peer model couldn't see.

Settling a noisy debate

A recurring claim in interpretability research is that you can train a small classifier, called a probe, on an LLM's hidden states and predict whether its answer will be right, sometimes even before it generates the answer. That would imply models carry an internal correctness signal, a kind of self-awareness. Recent work has pushed back, arguing that probes mostly detect retrieval activation patterns rather than correctness signals, and that external models can match the prediction performance of methods relying on a model's own internal representations.

The new work argues both camps were measuring the wrong thing. Prior conclusions about the absence of privileged knowledge may be premature due to confounded evaluation: when external models can exploit proxy signals from shared correctness patterns, genuine privileged knowledge, if it exists, gets masked.

The agreement trap

The core insight is deceptively simple. If Gemma and Llama tend to get the same trivia questions right and wrong, then a probe reading Gemma's activations can predict Llama's correctness just fine, not because Gemma sees inside Llama, but because both models fail on the same hard questions. Models agree on correctness for roughly 80% of questions in factual tasks and 75% in mathematical reasoning, which turns peer models into cheap proxies.