LM Arena Ships Agent Mode to Rank AI Models on Real Multi-Step Tasks

The team behind the LM Arena leaderboard, the de facto popularity contest for frontier models, just shipped a product that pushes its evaluation work past single-turn chat. Agent Mode turns Arena into a sandboxed agent runtime, and the traces it collects feed a new leaderboard built on causal inference rather than pairwise vote counts.

Agent Mode autonomously builds a plan and uses built-in tools to accomplish a multi-step workflow in one go, like building a website or running deep research, instead of forcing users to chain prompts. The toolset includes web search, image generation, coding and technical assistance, file attachments, and a sandbox bash environment for testing and iteration. Frontier models including GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro are wired up alongside top open-weights models.

From pairwise votes to causal tracing

The interesting part is not the chat wrapper, it is the evaluation method underneath. Rather than pairwise votes, rankings are calculated using a methodology called causal tracing, which treats the agent as a multi-component system where each component selection represents a possible treatment. By randomizing which orchestrator model, subagent, or harness piece gets used in each session, Arena turns live usage into a multi-intervention randomized controlled trial in which measurements can be aggregated to estimate causal treatment effects, which they call net improvement.

In plain terms: if you swap Claude Opus 4.7 in as the orchestrator while everything else is held statistically constant, how much does the success rate move? That is the number on the leaderboard, with 95% confidence intervals attached. This first leaderboard ranks orchestrator models, the main LLMs that choose which tools to call, with rankings of other harness components coming later.

The five signals

Arena combines five behavioral signals into the headline score. None of them require paid evaluators or curated prompts.

• Confirmed success, where the user marks a task success or failure using approve and disapprove buttons on every turn.
• Praise vs. complaint, identifying explicit verbal praise like "looks great" or complaints like "this is broken".
• Steerability, whether the agent successfully executes on in-line user corrections.
• Bash recovery, counting follow-up bash calls until the next non-erroring command after a model-induced error.
• Tool hallucination, penalizing invented tool names, malformed syntax, and chain-of-thought tokens leaking into the tool field.

There is also a cost axis. Arena calculates the exact post-deployment cost of a session and finds some models more expensive in practice despite cheaper on-paper pricing, as a result of model behavior such as more steps per turn, or induced user behavior such as more turns to reach satisfaction. If your agent picks a cheap model that then takes 40 tool calls to finish a job, that shows up.

What the trace stream actually shows

The methodology post drops a substantial dataset on how people are actually using agents. In a recent 7-day slice, Arena saw 160,480 Agent Mode tasks across categories including code writing at 17.5%, research and lookup at 10.8%, planning and brainstorming at 10.6%, multimodal image and video work at 10.2%, document creation at 9.1%, and code debugging at 8.9%.

Volume is non-trivial. Across 128,244 sessions, 75.6% used at least one tool, 41.1% ran bash, and 27.1% ran web search, with Agent Mode issuing 2 million structured tool calls in the week including roughly 936,000 bash calls, 550,000 file writes, and 275,000 web searches. Tracking via successful write_file calls, Agent Mode wrote 40.3 million lines of code in the last week, roughly 1,000 lines per coding session. Context budgets are getting chewed up too: about 32% of recent sessions ended with at least 128k input tokens in the final turn, 22% with at least 256k, and 8% with at least 1M.

Bluster and bluffing

Two findings from the trace mining are worth flagging because they describe failure modes that benchmarks rarely catch. The first is what Arena calls bluffing. When the opening ask bundles several explicit parts, agents usually cover all of them; the typical shortfall is leaving one incomplete, but a rarer and more consequential shortfall is covert: the agent could have surfaced the incomplete work, but instead presents the result as complete.

The second is bluster. Agents do sometimes push back against users, but they usually only sound firm, rarely holding their ground in practice, an artificial assertiveness that melts under additional pressure. Arena also observed a clear pattern in how people delegate: most opening messages hand over a whole deliverable, but after seeing the first response, users tighten the reins, pulling control back far more often than they hand over more.

Why it matters

Agent evaluation has been stuck between static benchmarks like SWE-bench and GAIA, which saturate, and vendor demos, which cannot be trusted. The Agent Arena methodology is the first public attempt to fold orchestrator choice, tool harness, and cost into a single causal estimate using live workload. If you are picking a model for an internal agent product, this is one of the few public signals tied to real users actually downloading the artifacts they asked for.

You can try it free at arena.ai/agent, and the orchestrator-only leaderboard is the first cut. Rankings for subagents, image generators, and harness components are next, which should make it easier to debug why your stack is losing on bash recovery without blaming the model itself.