Skip to main content
Back

Fine-Tuned AI Planners Know More Than Their Outputs Reveal

A new interpretability study reveals that fine-tuned LLM planners quietly build internal world models, and that training data diversity is the key to unlocking them.

Paper, Full Access
  • Fine-tuned LLM planners linearly encode action validity and world-state predicates in their hidden layers, even without explicit supervision on those properties.
  • A model can have strong internal world-model representations while still failing to express that knowledge through output token probabilities, making generation-based evaluation misleading.
  • Training on random-walk data (broad state coverage) produces significantly better world model recovery than training on optimal plans alone.
  • The paper introduces a reusable interpretability recipe: linear probing + generative chain-of-thought testing applied to planning LLMs.
  • Experiments run on Blocksworld and Logistics domains using the Plansformer architecture across three training data regimes.
  • No code or model weights released; findings are most relevant to teams fine-tuning LLMs for agentic or sequential decision-making tasks.

When you fine-tune a language model to solve classical planning problems, does it actually learn how the world works, or is it just memorizing action sequences? A new paper from Emami, Qiang, and Graf takes a surgical look inside fine-tuned LLM planners and finds something surprising: the models are building genuine internal representations of their planning environments, even when their outputs suggest otherwise.

The gap between doing and knowing

LLMs can achieve strong performance in planning through substantial fine-tuning, but fine-tuning is associated with significant economic and computational costs. The deeper question, which the community has largely left unanswered, is whether a model that learns to output valid action sequences is also learning the underlying rules of the world it is navigating. These are very different things. A model could memorize patterns in training plans without ever forming a coherent mental model of state transitions, preconditions, or effects.

There are active debates over whether LLMs are just memorizing vast amounts of statistics or if they can learn a more cohesive representation of the world whose language they model. This paper attacks that question head-on, specifically in the context of classical planning, where the rules are formal, verifiable, and unambiguous.

What classical planning actually demands

Classical planning, in the AI sense, is the problem of finding a sequence of actions that transforms an initial world state into a goal state. A classical planning problem is a pair consisting of a domain (predicate symbols and action schemas) and a problem instance (objects, initial state, and goal). A sequence of grounded actions is a valid plan if and only if it is executable and the goal holds in the final state after executing the plan. Standard benchmarks include Blocksworld (stacking colored blocks into target configurations) and Logistics (routing packages through cities using trucks and planes).

The challenge for LLMs is that valid planning requires tracking state: you need to know what is currently true about the world at every step to know which actions are legal. LLM-based world models are computational frameworks that use pretrained language models to simulate state transitions and reason with compositional, high-level world knowledge, integrating direct transition modeling, symbolic rule learning, and retrieval augmentation to enhance planning and decision-making.