Vals AI's Terminal-Bench 4.0 Shows Most AI Agents Fail 70% of Expert Tasks

Vals AI has published the Terminal-Bench 4.0 leaderboard, and most evaluated models failed at least 70 percent of its tasks. Only three systems cleared 30 percent on work designed to take a human expert roughly four hours per task.

A steep leaderboard

Below the top three, no model reaches 30 percent. Half the field scores 10 percent or less, and several models score zero. The results expose a large gap between agents that can solve isolated coding problems and agents that can finish long, multi-step jobs in a terminal.

A 66-task reset

Terminal-Bench is an open-source evaluation for agents operating inside a sandboxed terminal. Version 4.0 replaces the previous task set with 66 new tasks. None overlaps with the 89 tasks in Version 2.1, and the former easy, medium, and hard tiers have been replaced by domain categories.

Each task has its own verifier suite and uses all-or-nothing grading. The agent must pass every required test to receive credit for that task. Reported scores use avg@3, the mean of three independent single-attempt runs, with an eight-hour agent limit for each run. A score of 57.1 percent therefore represents complete success on about 57 percent of task attempts, with no credit for partially finished work.

Seven domains stretch beyond code

Software engineering accounted for most of Terminal-Bench 2.1. In Version 4.0, roughly three quarters of the tasks cover science, machine learning systems, operations, hardware, security, and media.

One operations task asks the agent to accelerate worker startup in a financial transaction pipeline so overdraft notifications arrive within five seconds during rolling deployments. The agent must modify the system, add any required Python packages to a specified requirements file, and satisfy the task’s verifier. That structure tests whether a model can complete a realistic operational change across diagnosis, implementation, and validation.

Version 4.0 also recalibrates compute and time allowances, revises instructions and environments, and tightens verifiers. The maintainers excluded eight candidate tasks because they were saturated, prone to refusals, publicly solved, or affected by unresolved quality problems. Those changes reduce obvious sources of benchmark artifacts.

Leaders change with the domain

• GPT-6 Astra leads five of seven categories, including media at 75 percent and hardware at 73.3 percent.
• Claude Opus 5 leads machine learning at 63.6 percent.
• Claude Fable 5.1 and Claude Opus 5 tie on security at 73.3 percent, ahead of Astra at 53.3 percent.
• Operations records the lowest field-wide mean at 11.2 percent.
• Fifteen models score zero on hardware, and 15 score zero on media.

Price and performance diverge

Reported inference spending per task does not reliably track benchmark performance. Several lower-cost runs outperform models that consume far larger budgets.

An eight-hour limit gives a stalled agent ample time to keep generating tokens and issuing commands. Developers evaluating agent economics should track convergence, retries, wall-clock time, and token use alongside the final pass rate.

Fallbacks blur model identity

Vals flags provider-side fallback in three Anthropic rows, meaning the gateway served some attempts with a model other than the one requested. Claude Fable 5.1 had 28 of its 198 attempts routed to Claude Opus 5. Its requested-model-only score was 42.4 percent, compared with the published aggregate of 49.5 percent.

Claude Fable 5 had 51 attempts handled by Claude Opus 4.8, and its requested-model-only score falls to 16.7 percent after separating those runs. These aggregate rows represent provider-routed services, so checkpoint comparisons should use the requested-model-only figures. Provider refusals count as failures; Qwen 3.8 Max recorded 21.

From benchmark to deployment

Terminal-Bench runs each model through Terminus 2, a minimal, model-agnostic agent harness. The model receives task instructions and access to a tmux session, emits keystrokes to execute terminal commands, and reads the resulting screen after each turn. A shared harness makes model comparisons cleaner by holding the interaction layer constant.

The environment resembles the shell, repository, specification, and long execution window found in coding agents such as Claude Code and Codex. Production systems may add planning loops, retrieval, specialized tools, retries, memory, and human approvals, so their completion rates can differ from the leaderboard.

Teams applying these results can reduce deployment risk through a few concrete steps:

• Evaluate models on representative internal tasks instead of relying only on the aggregate score.
• Compare category results when choosing models for security, operations, hardware, media, or machine learning work.
• Log the model that actually served each request and detect provider fallback.
• Set token, cost, and wall-clock limits for stalled runs.
• Require verifier checks and human approval for consequential changes.

The strongest configuration completes 57.1 percent of these all-or-nothing tasks, while operations averages 11.2 percent across the field. Current agents can support constrained workflows with strong verification, but multi-hour unattended work remains unreliable across many domains.