Arena.ai replaces Bash Recovery with Bash Reliability

Arena.ai has replaced Bash Recovery with Bash Reliability on its Agent Arena leaderboard. The update recalculates rankings for 52 frontier models using more than 2.5 million live agent sessions. Claude Opus 5.5 (Max) now leads the overall board, and Anthropic models hold four of the top five positions for shell reliability.

A stricter test for shell calls

Bash Recovery emphasized whether an agent could recover after a shell command failed. Bash Reliability evaluates the command that produced the outcome. A call counts as a failure when the agent causes the error, including through a mistyped flag or a defect in a generated script.

Correctly executed tests that uncover defects count as successful shell calls. Searches that return no matches also count as successes because the command ran as intended. Failures outside the agent’s control, such as network outages, are exempt from scoring.

Arena calculates Bash Reliability from agent-caused failures across all non-exempt Bash calls. A higher failure rate lowers Net Improvement, Arena’s aggregate leaderboard score. Models that issue valid commands on the first attempt avoid the penalty incurred by failed calls, even when another call later fixes the problem.

Proprietary models fill the top 14

Claude Opus 5.5’s score is based on 5,686 sessions. Proprietary models occupy the first 14 positions. The highest-ranked open-weight entries are GLM 5.2 and Xiaomi’s MiMo V2.6 Pro, both roughly 10 percentage points behind the overall leader.

Anthropic takes four shell spots

The Claude family has been tuned for tool use in Claude Code, including command-line workflows. GPT 6.1 Sol ranks second overall with a 2.63% Bash Reliability score. Overall Net Improvement combines five signals, allowing stronger results elsewhere to offset a lower Bash score.

What Arena counts as failure

Arena derives its signals from live user traces, including turns, corrections, errors, explicit confirmations and feedback. The label applied to each event therefore affects which behavior the leaderboard rewards.

Under Bash Recovery, a model that failed frequently could receive substantial credit by repairing its commands. Bash Reliability assigns the original agent-caused failure to the model, even when a later call succeeds.

Bash Reliability contributes to the overall ranking alongside Confirmed Success, Praise vs Complaint, Steerability and Tool Hallucination. Arena’s signal definitions explain how the metrics roll up into Net Improvement.

A tool call counts as a hallucination when it invents a tool name, uses a nonexistent signature or references an unavailable capability. Arena reports tool-hallucination rates below 1% for most frontier models. Mistral Large 4 is an outlier at 17.65%.

Match the model to the workload

For coding agents and shell-heavy automation, Bash Reliability offers a more relevant measure of agent-caused command failures. Deployment-specific tests remain necessary because operating systems, shell dialects, permissions, tool schemas and retry costs vary across environments.

1. Shell-heavy workloads: Anthropic’s models lead this signal and are the strongest candidates when failed calls or retries carry high costs.
2. End-to-end agents: Claude Opus 5.5 and GPT 6.1 Sol have the highest overall Net Improvement scores.
3. Open-weight deployments: GLM 5.2 and MiMo V2.6 Pro are the highest-ranked options, though both trail the proprietary leaders.
4. Local evaluation: Re-run representative commands with the production shell, permissions, tools and retry policy before choosing a model.

Arena says it will publish backfilled Bash Reliability scores in its public Hugging Face datasets, enabling developers and researchers to reproduce analyses with the revised metric.