Sakana AI gives coding agents a structured research loop

Sakana AI has released The Agentic Scientist, the third version of its automated research system. The open-source scaffold coordinates coding agents across hypothesis formation, experimentation, ablation, evaluation, and paper writing. It addresses a recurring problem with general-purpose coding agents: they often optimize a benchmark score without investigating the mechanism behind the improvement.

The release includes two Apache 2.0 repositories: AI-Scientist-v3 for the main pipeline and AutoIllustrate for editable scientific diagrams. An accompanying paper documents the architecture, case studies, and evaluation sweep.

Six stages, separate sandboxes

The Agentic Scientist decomposes a research campaign into six sequential phases. Each phase runs in its own Docker container and reads only a run-local snapshot, which limits cross-stage contamination and creates an inspectable record of the work.

1. Task drafting: Converts a task.md brief into an evaluator and scoring rubric, then seals the evaluation assets.
2. Investigation: Repeats a cycle of proposing ideas, implementing experiments, and evaluating results.
3. Ablations: Compares the selected program with baselines and variants that remove individual components.
4. Final evaluation: Scores the selected program again in a sealed environment.
5. Post-hoc analysis: Examines saved artifacts for additional evidence and failure modes.
6. Paper production: Writes and compiles the manuscript, then audits its claims against the recorded evidence.

Within each investigation worker, an ideation agent proposes a hypothesis, implementation plan, and rubric describing what the experiment must demonstrate. An execution agent edits candidate programs, runs evaluations, and records metrics. Parallel workers exchange memory at cycle boundaries, allowing successful techniques and failed approaches to inform later experiments.

Three case studies test the loop

Across three case studies, Sakana’s clearest external result came from research on Error Diffusion, a biologically inspired alternative to backpropagation. Work produced through the system led to a paper accepted for an oral presentation at ALIFE 2026. Human researchers chose the question, reviewed the evidence, and verified every result before submission; the agents handled much of the implementation and experimental work.

In a long-context retrieval task based on Doc-to-LoRA, the system found that combining document-specific LoRA adapters before receiving a query caused facts to interfere with one another. LoRA adapters are small sets of model adjustments that can encode information without retraining the full model. The resulting method selects the relevant adapter after seeing the query and refines both text selection and chunk boundaries. Sakana reports that exact-match accuracy rose from 52% to 98%, with no additional trainable parameters or retraining.

For a learned-optimizer task, the system developed a strategy based on CMA-ES, a gradient-free method that searches over candidate solutions. Learned adjustments improved performance over the course of optimization relative to the reference CMA-ES implementation. The system then ran ablations to identify which components produced the gain.

The model drove more variance

Sakana compared nine frontier language models across four agent frameworks. The model set comprised four GPT models, four Claude models, and one DeepSeek model; the harnesses were Codex, Claude Code, OpenCode, and Pi. The evaluation covered 70 research campaigns across five tasks.

Model choice had a larger measured effect than the agent harness. Newer models generally earned higher reviewer scores, especially within the Claude family. Numerical task performance and reviewer scores had a Pearson correlation of r = 0.44, indicating that the reviews captured some real performance gains while also reflecting criteria beyond the headline metric.

Editable figures stay editable

AutoIllustrate uses an editable draw.io vector diagram as the source of truth for each scientific figure. Pixel-based outputs are generated from that file, so researchers can revise shapes, labels, and layout by hand. A deterministic checker validates geometry and rejects overlapping elements before an image model applies the final visual style.

From installation to first run

Running the system requires uv, Docker, and credentials for the selected model provider. A minimal CPU-only run uses the bundled circpack example:

docker build -t agentic-scientist/agent-cli:latest -f src/docker/agent-cli/Dockerfile .
uv sync && source .venv/bin/activate
agsci doctor examples/circpack
agsci run examples/circpack

The default agent is Claude, and each phase runs inside Docker. Developers can select another harness with an override such as agent.name=codex or agent.name=opencode. GPU workloads, including GRPO reinforcement-learning fine-tuning, have separate presets.

The README warns that coding agents receive full tool permissions within their worker environment through headless “allow yolo” mode. They can execute arbitrary commands, access supplied API keys, and follow instructions embedded in task files or papers. Runs should remain inside the provided containers, with credentials and mounted files limited to what each experiment needs.

Shortcuts remain a live risk

A high evaluation score can still come from a shortcut that violates the intended method, a failure mode commonly called reward hacking. An early Error Diffusion run found such a shortcut because important constraints were implicit, prompting Sakana to make the rubric more explicit.

• Late literature search: The system searches published work while writing the manuscript, after hypotheses have already been generated.
• Weak baseline generation: The pipeline cannot reliably turn published methods into competitive implementations, so comparisons focus mainly on variants of the selected program.
• Narrow domain evidence: All reported tasks involve computational machine learning. Performance in physical experiments, instrument-dependent research, and human-subject studies remains untested.
• Benchmark saturation: On mature benchmarks, the search can collapse into hyperparameter tuning instead of exploring new mechanisms.

Sakana describes the system as “co-discovery”: humans define the research question, evaluation rubric, and required standard of evidence, while agents run more experiments and preserve the resulting artifacts. For teams already using Claude Code or Codex in research engineering, the release provides a reproducible scaffold for turning agent activity into hypotheses, ablations, sealed evaluations, and evidence-linked manuscripts.