Prime Intellect's Fable 5 Closes 82% of the Human AI Research Gap
Prime Intellect ran 153 autonomous research runs across 18 frontier models on the nanoGPT optimizer speedrun, with the best closing 82% of the gap to a record built by dozens of humans over months.
- Prime Intellect ran 153 autonomous research runs across 18 frontier models on the nanoGPT optimizer speedrun, the largest open experiment of its kind.
- Best runs closed 82% of the gap to a human record built by dozens of researchers over months; Fable 5 led at 81.7% gap closure (2,726 steps).
- Top models succeeded by selecting the right experiments, navigating benchmark noise, and revisiting old negatives as the recipe evolved.
- Key finding: frontier agents excel at optimizer search and hyperparameter sweeps but struggle to generate genuinely novel ideas without upstream human records.
- All traces, scratchpads, and reasoning streams are publicly released, including 41 curated full agent trajectories with tool calls and subagent behavior.
- Prime Intellect plans to extend speedruns to more of the training stack and use multi-agent harnesses with smaller open models to reduce compute costs.
Prime Intellect has published what it calls the largest open experiment on autonomous AI research: 153 autonomous runs across 18 frontier models, all competing on the same well-defined ML optimization task. The goal was not to crown a winner, but to measure how capable today's frontier models actually are at doing research autonomously, and to understand where they succeed and where they fall apart.
The arena: a constrained, measurable research task
The benchmark is the nanoGPT optimizer speedrun, a public leaderboard where the objective is to train a 124M-parameter GPT model to a fixed validation loss in as few steps as possible. The goal is simple: lower the number of steps needed to reach a target validation loss while only changing the optimizer, schedules, initialization, and some hyperparameters. No internet access. No changing the architecture. Just optimizer research, sandboxed on 8xH200s for up to 8 days per run.
This constraint is what makes it a meaningful proxy for research ability. The search space is large enough to require genuine hypothesis generation, but the feedback loop is tight enough to measure progress objectively. A human community built the current record over months of collaborative work, giving a concrete ceiling to measure against.
What they tested, and how
Prime Intellect ran every model inside their Prime Agent harness, which gives each model a persistent IPython kernel rather than a flat tool menu. Prime Agent treats context as a variable with programmatic access through a persistent IPython REPL, enables agent-to-agent communication and persistent sub-agents, and implements self-improvement through trajectory-based refinement. This means variables and state survive across turns, and models can build their own tooling mid-run rather than being constrained to pre-defined actions.
The 18 models tested included:
- Fable 5, Opus 5, Opus 4.8, Sonnet 5 (Anthropic)
- GPT-5.6 Sol, GPT-5.6 Sol Pro, GPT-5.6 Luna, GPT-5.6 Terra, GPT-5.5 (OpenAI)
- Kimi K3, Kimi K2.7 (Moonshot)
- Grok 4.5, Grok 4.6 (xAI)
- GLM 5.2, GLM 5.3 (Zhipu)
- DeepSeek V4 Pro
- Qwen 3.8 Max
- Muse Spark 1.1, Muse Spark 1.2
All full traces, scratchpads, and reasoning streams from open-weight models are publicly released alongside the experiment setup.
The results: a wide spread, with a clear frontier
The leaderboard shows 153 autonomous runs across 18 frontier models, with the best runs closing 82% of the gap to a human record built by dozens of researchers over months. But the spread between models is enormous. Here is how the top performers stacked up:
| Model | Best Record (steps) | Gap Closed | Run Duration |
|---|---|---|---|
| Fable 5 | 2,726 | 81.7% | 8.7 days |
| Opus 5 | 2,920 | 53.6% | 2.9 days |
| Kimi K3 (Prime Agent) | 2,930 | 52.2% | 3.6 days |
| Kimi K3 (Kimi Code) | 2,974 | 45.8% | 5.1 days |
| GPT-5.6 Sol | 3,042 | 35.9% | 6.1 days |
| Grok 4.5 | 3,120 | 24.6% | 2.7 days |
| GLM 5.2 | 3,150 | 20.3% | 1.8 days |
| Muse Spark 1.1 | 3,232 | 8.4% | 3.7 days |
The baseline the human community started from is 3,290 steps. The human record sits at 2,600. Fable 5 reached 2,726, Opus 5 reached 2,920, and the bottom of the pack (Muse Spark 1.1, GPT-5.5) closed less than 10% of that gap. That is not a small difference -- it is a qualitative gap in research behavior.
What separated the strong from the weak
Prime Intellect's analysis points to three distinguishing behaviors among the top models:
- Experiment selection: Knowing which hypotheses are worth a full GPU run versus which can be ruled out analytically.
- Noise navigation: The nanoGPT benchmark has inherent variance. Weaker models chased noise; stronger ones ran controlled comparisons and built statistical intuition.
- Revisiting old negatives: As the recipe evolved, some previously-failed ideas became viable. The best models tracked this and looped back.
Some models went further. Kimi K3, for instance, built its own tooling mid-run: tools for controlled optimizer variants, loss-curve comparisons, and Newton-Schulz tuning, then revised its hypothesis when a cleaner update lost in training. Some models even built small simulations to isolate a mechanism before deciding if another GPU run was worth it -- behavior that looks qualitatively similar to how a careful human researcher would work.
The real finding: agents are good at search, not invention
The headline gap-closure numbers are impressive, but the more important finding is about the shape of that capability. Agents are very good at optimizer search, hyperparameter sweeps, and stacking methods together, but they struggle to come up with new ideas on their own and need upstream human records to keep improving.
This matches what METR found in a separate evaluation. Although some models have expenditure horizons in the thousands of dollars, they are small relative to the overall human labor, indicating that autonomous agent optimization has so far had minimal effect on AI R&D progress in NanoGPT. The speedrun's maintainer estimates roughly 70% of ideas would be mergeable, but what the agent mostly explores (e.g., hyperparameter tuning) is of low novelty.
The community has also pushed back on framing this as "AI research" in the full sense. We need to be careful about claiming improvements on nanoGPT speedrun optimizer would correspond to truly better full on AI research. The task is real and the results are meaningful, but it is a constrained optimization problem -- not open-ended scientific discovery.
What this means for the field
The practical implication is that autonomous agents are already useful as a force-multiplier for human researchers on well-defined optimization problems. The nanoGPT track is a proxy for a class of real work: hyperparameter search, optimizer ablations, training recipe iteration. For that class of task, the best frontier models running for days on dedicated compute can now cover ground that would take a human team weeks.
Prime Intellect also flagged where they think the setup can improve. Multi-agent harnesses using smaller open models for monitoring and implementation could make these experiments significantly cheaper. They also plan to extend speedruns to more of the training stack beyond just the optimizer, and to scale the runs themselves.
Everything from this study is publicly available: full traces, scratchpads, reasoning streams from open-weight models, and the experiment setup. For anyone building autonomous research pipelines, the 41 curated full agent trajectories -- including tool calls, subagent spawns, and scratchpads -- are the most useful artifact to study.
Comments
