Ai2's StateBench Catches Frontier Models Failing at 13% on Long State Tracking

Ai2 has released StateBench, an open benchmark for testing whether language models can preserve and update information across long inputs. In Ai2’s evaluations, frontier models performed well on simple lookup tasks but lost accuracy when later updates depended on earlier state. Accuracy on the longest general finite problems fell to 13% or lower.

State tracking requires a model to compute the current value after a sequence of changes. Examples include following a reassigned variable through a codebase, updating a running score, or identifying who holds an object after several handoffs.

Four tiers, one controlled trace

StateBench organizes tasks into four complexity tiers derived from Krohn-Rhodes algebra, a framework for classifying finite state-transition systems. Ai2 maps each tier to an assignment pattern familiar to programmers:

Each benchmark family begins with a general finite sequence and derives three easier versions from the same trace. Variable counts and assignment positions remain fixed while the assignment operation changes, allowing comparisons that control for sequence structure and length.

Length breaks frontier performance

Ai2 evaluated four frontier models: Gemini 3.8 Flash, Claude Opus 5.5, and two variants of GPT-6 (Sol and Luna). Each model read a sequence of assignments and returned the final value of every variable.

Performance remained strong on R-trivial and aperiodic tasks. Accuracy fell sharply on long soluble and general finite sequences, where more assignments can influence the answer. A missed update can corrupt every later value that depends on it.

Recurrence changes the curve

To isolate architectural effects, Ai2 trained several 275-million-parameter models from scratch on StateBench sequences containing up to 256 assignments. The group included transformers, recurrent Gated DeltaNet variants, and hybrids combining recurrent and attention layers. Evaluation sequences extended to 64,000 assignments, 250 times the maximum training length.

• R-trivial, aperiodic, and soluble: Gated DeltaNet and hybrid models remained near perfect through sequences containing tens of thousands of assignments.
• Transformer extrapolation: Transformers learned the easier tasks but degraded as evaluation sequences exceeded the training range.
• General finite: Transformers did not solve this tier in the controlled experiment. The more flexible Gated DeltaNet variant stayed near perfect to roughly 2,560 assignments, about 10 times its training limit.

Successful task-specific training shows that recurrent and hybrid architectures can represent the first three tiers and extrapolate far beyond their training lengths. Weak performance from pretrained recurrent models can therefore plausibly reflect training data or optimization. The transformer results provide evidence of an architectural constraint within this setup, although experiments on 275-million-parameter models do not establish a universal ceiling.

The proof draws a boundary

The paper proves that a constrained Gated DeltaNet can solve every aperiodic task and some periodic tasks under fixed-precision arithmetic with rounding. The constraint requires non-negative eigenvalues in the memory update, limiting how the model’s hidden state can evolve.

The covered periodic tasks include parity tracking, such as determining whether an input has appeared an even or odd number of times. The model’s exact position in the hierarchy remains unresolved, as does the effect of greater numerical precision or additional attention layers.

One trace, several disguises

The Hugging Face dataset is generated procedurally from canonical assignment traces. Each trace can be rendered as integer code, spreadsheet edits, cube painting, status lights, or people moving between rooms.

Ai2’s format-sensitivity test found that small hybrid and Gated DeltaNet models scored near perfectly on RULER-style R-trivial tasks, which resemble long-context retrieval tests. The same underlying traces produced near-chance scores when presented as spreadsheet edits. Testing several surface forms helps separate state-tracking ability from familiarity with a specific prompt format.

Running the benchmark

StateBench uses YAML configuration files to control the main evaluation variables:

• Assignment depth: Run logarithmic sweeps up to 64,000 assignments.
• Distractors: Add easy, medium, or hard no-op filler lines.
• Token length: Group examples into buckets such as tokens_64k, tokens_128k, tokens_512k, and tokens_1m.
• Surface form: Render equivalent traces in several task formats.

After setting up the repository, the following command samples a reproducible periodic integer-code problem:

uv run state-bench sample \
  configs/experiments/default.yaml \
  integer-code--periodic \
  --num-variables 4 \
  --extra-assignments 1024 \
  --seed 7

The command creates a case with four variables, 1,024 extra assignments, and a fixed random seed. Researchers can vary those arguments to measure performance by dependency depth, context length, and representation.

Where the results apply

Many long-context benchmarks emphasize retrieval, where the model locates information that already appears in the input. StateBench measures ordered computation over changing values, where the answer emerges only after relevant updates have been applied.

The benchmark maps directly to several developer workloads:

• Coding assistants that follow variables across large files or execution traces
• Agents that maintain state across long sequences of tool calls
• Systems that track entities, permissions, or commitments through extended conversations
• Workflow models that process event logs with dependent updates

StateBench’s synthetic, deterministic traces isolate one capability and make failures reproducible. They do not cover every form of long-context reasoning, and the architecture study’s parameter scale limits conclusions about larger systems. Production evaluations should match the dependency depth, input format, and error costs of the intended workload.

For applications dominated by long chains of dependent updates, the results support testing recurrent and hybrid architectures alongside pure transformers. Ai2’s full analysis includes results for open OLMo 3 and OLMo Hybrid models, plus the 275-million and 2.7-billion-parameter variants used in the architecture comparison.