Skip to main content
Back

Google DeepMind Says Transformers Are Geometrically Broken at State Tracking

A Google DeepMind paper argues transformers are architecturally incapable of true state tracking, and makes the case for a return to recurrence

Pro

Paper, Full Access
  • Core claim: Transformers cannot do true state tracking because their feedforward topology forces each state update to live deeper in the layer stack, eventually running out of depth.
  • The failure mode: Models like Gemini flip-flop on word meanings and game rules mid-conversation because shallow layers processing new tokens can't access disambiguations made in deep layers earlier.
  • Chain-of-thought is a workaround: Extended thinking externalizes state into the context window, which works but wastes compute and memory for updates that should be automatic.
  • New taxonomy: The paper maps all recurrent/hybrid architectures by two axes -- recurrence direction (depth vs. step) and tokens-per-recurrence-step ratio -- clarifying which can achieve true indefinite state tracking.
  • Best candidates: RWKV-7 and gated linear attention hybrids are highlighted as achieving greater expressivity than transformers while remaining parallelizable during training.
  • No code released: This is a position/survey paper from Google DeepMind; it provides a research roadmap, not a new model or implementation.

Every time a frontier model loses track of who said what in a long conversation, forgets the rules of a game it agreed to play, or flip-flops on the meaning of an ambiguous word, it is exhibiting the same underlying failure. A new paper from Google DeepMind titled "The Topological Trouble With Transformers" argues this is not a training problem or a data problem. It is a geometry problem baked into the transformer architecture itself.

The Depth Trap

Transformers' purely feedforward architecture fundamentally limits dynamic state tracking -- the iterative updating of latent variables reflecting an evolving environment -- because it involves inherently sequential dependencies that feedforward networks struggle to maintain. The authors, Michael Mozer, Shoaib Ahmed Siddiqui, and Rosanne Liu, give this a precise geometric interpretation: every time a new token arrives, the model's internal "belief" about the world must be recomputed, and because information can only flow upward through the layer stack, each new state update must live deeper in the network than the previous one.

Feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth. Think of it like a stack of sticky notes: every time you want to update a fact, you have to write it on a higher note. Eventually you run out of notes, and earlier updates become unreachable to the layers processing the next token.

Diagram showing how state representations shift upward through transformer layers with each new input step

When Models Lose the Plot

The paper illustrates the failure with two vivid examples. In one, Gemini 3 (Fast) is asked to play a number-guessing game. The model responds "lower" when the guess is 50, then "higher" when the guess is 51 -- a logical impossibility that reveals it never maintained a consistent internal state. In another, Gemini 2.5 Flash correctly identifies that "Fred" went fishing at a river bank, then immediately assumes he would find an ATM there when asked a follow-up question.

State tracking failures in foundation models lead to loss of coherence in multi-turn conversations, inefficiency in information gathering, and breakdowns in communication and cooperation in multi-agent settings. The paper cites research showing that the word "bank" gets disambiguated correctly only at deep layers of the network. But by the time the model processes the follow-up question, those shallow layers -- which handle the new token -- have no access to that deep disambiguation. Without proper state tracking, models flip-flop in their interpretations and fail to detect their inconsistencies -- and while newer models are less susceptible, the example reveals a fundamental limitation of the core architecture.