Stanford's Terminal-Bench-Science Exposes GPT-6 Astra Leading at 63% on Real Science Tasks

A new agentic benchmark tests frontier models on messy, multi-step scientific workflows using real data and research tools. The benchmark leaderboard shows that only GPT-6 Astra and Claude Opus 5.5 exceed 50%, leaving substantial room for improvement in end-to-end research automation.

GPT-6 Astra at max effort leads with a 63.3% pass rate. Claude Opus 5.5 reaches 61.9% at xhigh effort and 59.0% at max effort. The strongest open-weight models remain near 10%.

Seventy tasks, all-or-nothing grading

Terminal-Bench-Science is an open academic collaboration led by Stanford researchers, the Terminal-Bench and Harbor teams, and contributors from scientific institutions worldwide. Release 0.1.0 contains 70 tasks across five domains:

• 19 life science tasks
• 17 physical science tasks
• 17 mathematical science tasks
• 9 engineering science tasks
• 8 earth science tasks

Each task places an agent in a sandboxed terminal with instructions, data, and relevant software. Pytest-based checks grade the resulting files and calculations, and a task counts as passed only when every test succeeds.

Scoring uses pass@1, meaning one attempt per run, averaged across three runs for each task. Every model runs through the mini-swe-agent harness, which reduces variation from custom agent scaffolding and makes model and reasoning-effort comparisons more consistent.

The workflows require agents to combine data processing, scientific reasoning, tool use, and output validation. Examples include:

• Reconstructing lake freeze-up and ice-off dates across ten winters from raw satellite imagery, including implementation of a compound ice-cover criterion.
• Tracking every cell division in an 800-frame phase-contrast recording of mouse myoblasts, then labelling generations and outcomes such as division, death, fusion, or departure from the field of view.
• Calculating the geocentric state vector of an unknown near-Earth asteroid from raw Minor Planet Center astrometry, JPL ephemerides, and observatory metadata, with kilometre-scale position tolerances.

Each run may use up to 28,800 seconds, or eight hours, of wall time. The rules prohibit retrieving existing annotations or online solutions.

More inference helps, unevenly

Domain results vary sharply, although the benchmark authors caution that samples of 8 to 19 tasks produce noisy comparisons. Life science yields the weakest results for most leading models. Claude Opus 5.5 at xhigh effort passes 71% of mathematical science tasks and 46% of life science tasks.

Higher reasoning budgets produce large gains for some configurations. Claude Opus 5.5 rises from about 24% at low effort to 61.9% at xhigh effort, while average cost per task increases roughly fivefold. GPT-6 Sol gains 27 percentage points between low and max effort at approximately 7.5 times the cost.

Additional inference does not produce a consistent gain at every setting. Claude Opus 5.5 scores 59.0% at max effort, 2.9 points below its xhigh result. Developers choosing an effort level therefore need to consider measured accuracy and cost instead of assuming the largest budget will perform best.

Open weights trail by 50 points

GLM-5.3 and DeepSeek V4.1 Flash, the leading open-weight entrants in the release, score more than 50 percentage points below GPT-6 Astra. Qwen3.8 Max is the strongest closed model outside OpenAI and Anthropic at 12%, while Fable 5.1 trails the two leaders by roughly 20 points.

Why labs are reporting these scores

Many scientific evaluations measure factual knowledge through isolated questions. Terminal-Bench-Science measures whether an agent can complete an entire computational workflow, preserve intermediate state, recover from tool errors, and produce outputs that satisfy deterministic checks.

Frontier labs began publishing results within a week of the 0.1 release, and the benchmark appeared in the Claude Fable 5.1 and GPT-6 Astra release notes. Its all-or-nothing tests expose failures that partial-credit or model-graded evaluations can obscure, including small formatting, calculation, and data-processing errors that invalidate a research result.

The maintainers plan regular releases to keep the task set aligned with current model capabilities. Domain experts contribute workflows, technical reviewers convert them into reproducible terminal tasks, and model developers can use the resulting failures to target agent and tool-use improvements.

Run the suite or add a task

Developers can inspect the tasks and harness in the open-source GitHub repository. The benchmark provides useful evidence for teams that are:

• Building science-focused agents and testing whether terminal workflows generalise beyond software engineering.
• Comparing reasoning-effort settings against their accuracy and inference costs.
• Evaluating open-weight and hosted models under a shared agent harness.
• Contributing domain workflows that resemble real laboratory or computational research.

The project is accepting tasks for release 0.2 and lists October 5 as the submission deadline on its contribution page. Accepted contributors receive co-authorship on the planned paper, and each proposal undergoes parallel domain and technical review before it is merged.