Skip to main content
Back

Anthropic and Amsterdam Catch Qwen-3 Planning Ahead Before It Speaks

New mechanistic interpretability work on Qwen-3 shows LLMs form internal representations of future words, and this planning ability scales with model size.

  • New ICLR 2026 paper defines latent planning and traces it inside Qwen-3 (0.6B-14B) with feature circuits.
  • Only 14B+ Qwen-3 models reliably solve a/an agreement tasks that require knowing a future word like "accountant".
  • Feature circuits reveal explicit planning features representing future words that upweight the correct article.
  • Even 4B-8B models that fail contain nascent planning features, suggesting a gradual, scale-driven emergence.
  • On rhyming couplets, Qwen-3 does forward planning but almost no backward revision of prior context.
  • Code and analysis tools are available at github.com/hannamw/model-planning-public.

When a language model writes coherent prose or working code, is it actually planning ahead, or generating one token at a time and hoping for the best? A new ICLR 2026 paper from Michael Hanna (University of Amsterdam) and Emmanuel Ameisen (Anthropic) offers mechanistic evidence, finding that Qwen-3 models use planning mechanisms that scale with size.

The authors define latent planning as an internal representation that (1) causes the generation of a specific future token or concept, and (2) shapes preceding context to license that future token. The second clause is the interesting one. It separates real planning from a model retrieving a likely next token, because the network has to actively adjust what comes before to make the future word fit.

The a/an giveaway

The cleanest test uses English article agreement. Consider the prompt "Someone who handles financial records is a/an (accountant)". To output "an" rather than "a", the model must already have "accountant" in mind before emitting the article, because the choice depends on the following word starting with a vowel sound. Testing five Qwen-3 sizes from 0.6B to 14B, only models with 14B+ parameters consistently succeed.

To look inside the models, the authors use feature circuits, an interpretability technique that traces which sparse features (interpretable directions in activation space) causally drive a prediction. They find planning features that represent future outputs like