Epoch's EBR-Bench Shows GPT-5.5 and Claude Can't Learn From Practice

EBR-bench, a new benchmark from Epoch AI, asks a deceptively simple question: can today's best AI models actually get better at something through practice? After having GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro play a complex board game up to 30 times in a row, the answer is a clear no. The models show no meaningful improvement across repeated playthroughs -- a finding with real implications for how we think about AI autonomy and safety.

Why this question matters right now

An AI system that could pick up unfamiliar tasks on the fly would be much more capable than we're used to. Even if it didn't perform well out of the box on some economically relevant task, it could still learn "on the job." That's the economic upside. The safety concern is the flip side: it would also be harder to determine whether a model had dangerous capabilities prior to release, since it could gain such capabilities through learning.

This is one of the most actively debated open questions in AI capabilities research right now. Inference-time compute scaling -- the idea of spending more compute at runtime to get better results -- has shown real promise on reasoning tasks. But EBR-bench is asking something different: not "can you think harder on one problem?" but "can you accumulate knowledge across many attempts at an unfamiliar task?"

The game as a test bed

Earthborne Rangers (EBR) is a campaign-style game where a player explores a wilderness landscape, overcoming obstacles and pursuing objectives. It's relatively obscure, almost entirely card-based with very little spatial reasoning, and requires a mix of strategy and tactics around deck-building and turn-by-turn play. A single playthrough of the segment used in the benchmark takes humans 2 to 4 hours, and mastery can require dozens of playthroughs.

The choice of game was deliberate. To avoid AI systems being familiar with the games from training, it's useful that they be somewhat obscure. Since AI systems are already fairly capable at a wide range of one-off puzzle games, it's useful for the game to require a layered mix of strategy and tactics and to play out over a relatively long time horizon. Text-based games are also preferred because multimodal weaknesses shouldn't be the bottleneck -- the researchers want to isolate the learning failure specifically.

How the benchmark works

In the setup, AI systems play through the game many times (10 or 30, depending on settings). They are given the rulebook, a card database, and the game's map of locations. They are told their goal is to maximize the score they achieve on the final 20% of playthroughs, and are encouraged to take notes and learn from mistakes. A persistent note-taking tool carries information across sessions, meaning the model can, in principle, record what went wrong and act on it next time.

The scoring is out of 21 objectives. Expert humans score around 20. The models are evaluated on whether their scores trend upward over repeated play -- the core signal for on-the-fly learning.

Flat lines across the board

Frontier AI systems aren't terrible at the game. Their scores have improved modestly over the past year, with GPT-5.5 and Opus 4.8 outscoring GPT-5 and Opus 4.1. However, this is due to higher initial scores, not on-the-fly learning. There is no evidence that the ability to learn on the fly has improved at all over this time period.

Two specific failure modes explain most of the performance gap:

• Tactical failures (fatigue management): The primary tactical mistakes involve an in-game mechanic called "fatigue", which is similar to "damage" in other games. In Earthborne Rangers, minimizing fatigue is key to success, as it simultaneously represents the player's health bar, remaining time, and card resources. An expert human player can keep average fatigue per round to about 0.6. Completely random gameplay incurs 3.5 fatigue per round. AI systems are closer to the random baseline, with GPT-5.5 and Opus 4.8 both incurring 2.1 fatigue per round.
• Strategic failures (deck-building): Before a playthrough begins, players build a 24-card deck. A deck's power level can vary drastically -- the maximum score achievable by the strongest deck might be twice that of the weakest deck. Coarsely measuring deck types, there are 32 deck archetypes. Agents drastically underexplore the deck variety available, and there is no clear trend showing that more recent models explore more diverse deck archetypes.

Even handing them the answer key doesn't help much

To test whether the problem was note-taking quality rather than learning ability, the researchers ran a "max elicitation" condition: they gave models a complete, expert-written strategy guide covering all tactical and strategic advice. Comparing this setting to the basic setup, they saw modest improvement -- between 2 and 3.5 additional objectives (out of 21), depending on the model. But, considering that this approach essentially hands models the answer key, the effect is considered modest.

What this actually tells us about AI

This suggests that inference compute scaling doesn't overcome every capability deficiency. The returns to repeated play for an out-of-distribution game appear minimal. Even when there is a concrete metric -- like the number of objectives completed in EBR -- it seems that AI systems cannot always optimize their way toward it.

Since it seems plausible that focused training could improve performance on EBR and similar games, it's likely that AI companies simply haven't prioritized doing so. The findings are interpreted as evidence that AI systems remain fairly limited by their training distributions. In other words, the ceiling isn't raw intelligence -- it's the boundary of what the model has seen before.

What comes next

The researchers are candid that elicitation gaps may remain. Future experiments they plan to run include:

• Providing more tools like code execution or live web search
• Trying different agent scaffolds beyond the simple ReAct harness used here
• Multi-agent setups where models can consult each other
• Providing expert human playthrough transcripts as demonstrations

Some of these were already tested informally -- Claude Code and Codex with code execution enabled showed no significant improvement -- but the team says more rigorous experimentation is warranted. The EBR-bench will continue to be run against new model releases, serving as an ongoing tripwire: if and when on-the-fly learning emerges, this benchmark should be one of the first places it shows up.