Google Research Finds WikiProfile Reveals Recall, Not Knowledge, Breaks Frontier LLMs

When a large language model gets a fact wrong, the instinctive fix is to train it on more data or scale it up. A new paper from Google Research challenges that assumption. The real culprit is often a lost key, not an empty shelf.

When LLMs get facts wrong, the failure could stem from never learning them or from being unable to recall what's already encoded. The knowledge profiling framework introduced in this work points to the latter: frontier LLMs encode nearly all facts, yet struggle to retrieve many of them. That distinction reshapes how the field should approach factuality.

The diagnosis problem hiding in plain sight

Factuality is essential for making LLMs reliable. When a model answers incorrectly, the fact may have never been encoded, or it may be encoded but inaccessible. Standard accuracy metrics collapse these cases together, even though they imply very different limitations and very different interventions.

Encoding failures call for scaling model size or expanding data coverage. Recall failures point toward post-training and inference-time methods that help LLMs better use what they already store. Treating both as the same problem means applying the wrong fix, like buying more groceries when the issue is that you can't find what's already in the fridge.

A new lens: knowledge profiling

The paper introduces knowledge profiling, a behavioral framework that analyzes factual behavior at the level of individual facts rather than individual questions. Instead of asking "did the model answer this question correctly?", it asks "what is the state of this fact inside the model?" Each fact gets classified into one of five profiles:

• Encoding failure: the fact was never stored in the model's weights
• Recall failure: the fact is encoded but the model can't retrieve it
• Direct recall: the model answers correctly without any extra prompting
• Recall with thinking: the model needs chain-of-thought or extended reasoning to surface the fact
• Inference without encoding: the model derives the correct answer from related facts, even without directly encoding the target fact

To operationalize these profiles, the framework uses three behavioral probes. Encoding is tested by placing the model in a pre-training-like context, asking it to complete a sentence the way it would have seen the fact during training, without giving away the answer. Knowledge is tested by asking semantically equivalent questions in multiple phrasings, including both direct questions ("What is B?") and reverse questions ("What is A?" when A is B). Recall is then defined as successfully knowing a fact that has been confirmed as encoded.

WikiProfile: the benchmark powering this analysis

To support such profiling, the researchers built WikiProfile, a new benchmark constructed via an automated pipeline with a prompted LLM grounded in web search. The pipeline extracts facts from Wikipedia pages as ordered entity pairs (subject, object), then generates ten tasks per fact: two for encoding, four for knowledge evaluation, and four multiple-choice recognition variants.

Questions go through a three-step process of generation, refinement, and filtering to ensure each one is unambiguous, specific, and has a unique answer. After automated filtering grounded in a search engine and a final manual validation pass, the benchmark contains 2,150 facts. The team then evaluated 13 LLMs both with and without thinking modes, sampling eight responses per model-fact-task combination, producing roughly 4.5 million total responses.

Encoding is nearly saturated

Across those 4 million responses, encoding is nearly saturated in frontier models on the benchmark, with GPT-5 and Gemini-3 encoding 95–98% of facts. These models have essentially learned almost everything WikiProfile contains. And yet they still get facts wrong at a significant rate.

Recall remains the major bottleneck: many errors previously attributed to missing knowledge instead stem from failures to access it. These failures are systematic and disproportionately affect long-tail facts and reverse questions. For Gemini-3-Pro and GPT-5, even with near-perfect encoding, models still fail to directly recall 26–34% of facts. With thinking enabled, they still miss 11–12%.

The scaling story is equally revealing. In the Gemma 3 family, larger models show far fewer encoding failures, but recall failures remain substantial and actually become a larger share of the remaining errors. Scaling improves what the model stores more than it improves what the model can access.

Two systematic recall failure patterns

The paper identifies two specific conditions where recall breaks down predictably:

1. Rare (long-tail) facts. Prior work framed the long-tail problem as a capacity issue, assuming rare facts simply weren't learned. Here, rare facts are encoded at rates close to popular ones. The encoding gap between low-popularity and high-popularity facts is narrow; the recall gap is wide. The bottleneck is access, not storage.
2. Reverse questions and the reversal curse. The "reversal curse" is a known phenomenon where models trained on "A is B" can't answer "What is A?" when given B. This paper reframes it: in open-ended generation, reverse questions are consistently harder. In multiple-choice verification, reverse questions are no harder than direct ones, and often easier. If the model can recognize the correct answer among distractors but can't generate it, the bidirectional knowledge exists. That points to a recall problem rather than a knowledge gap.

Thinking as a recall recovery mechanism

One of the more actionable findings involves chain-of-thought and thinking-optimized models. Thinking improves recall most strongly exactly where direct recall is weakest, for rare facts and reverse questions, narrowing both gaps. Since thinking can recover a substantial fraction of encoded-but-not-directly-known facts, the next improvements in factuality may come from better utilization of knowledge already in the model, alongside better acquisition.

In thinking-optimized models, thinking recovers roughly 40–65% of encoded-but-not-directly-known facts. It helps much less, only 5–15%, on facts that are not encoded at all. Thinking primarily acts as a recall-facilitation mechanism: it surfaces what the model already knows, rather than deriving entirely new answers through complex reasoning chains.

What this means for practitioners

This work reframes where to invest effort in improving LLM factuality:

• Don't just scale. If encoding is already near saturation, further gains in factuality will come less from scaling model size or data. Throwing more compute at a recall problem won't fix it.
• Use thinking strategically. Enabling chain-of-thought or thinking modes is most valuable for rare facts and reverse-direction queries, exactly the cases where direct recall breaks down.
• Diagnose before intervening. Knowledge profiling enables precise diagnosis of factual behavior in LLMs. For a factuality-critical application, the right mitigation depends entirely on whether errors are encoding failures or recall failures.
• Reconsider the reversal curse. For applications that query facts in non-canonical directions (e.g., "who wrote this lyric?" rather than "what did this artist write?"), recall-boosting techniques like thinking or retrieval augmentation will likely outperform retraining.

The paper is available on arXiv, and the WikiProfile benchmark is publicly released on Hugging Face. No model weights or fine-tuning code accompany the release; this is a diagnostic framework and dataset, not a new model. The full Google Research blog post includes additional breakdowns across all 13 evaluated models.