Artificial Analysis' Speech Arena Reveals Voice AI's Uncomfortable Split Brain Problem
Artificial Analysis launched a blind human-preference leaderboard for voice agents, and the model users like most is not the one that finishes the task.
- Artificial Analysis launched the Speech Agent Arena, blind human preference for voice agents on 35 real tasks.
- Gemini 3.1 Flash Live Preview - Minimal leads preference at 1046 Elo but only 74.6% task success.
- Grok Voice Think Fast 2.0 High leads task completion at 94.7%, ranks ninth on preference.
- Preference tracks responsiveness closely: lower Time to First Audio correlates with higher Elo.
- Prices span $1.50 to $10.75 per hour of input audio across leaderboard models.
- Arena replaces Conversational Dynamics in the Speech to Speech Index at 25% weight.
Voice agents have quietly become one of the messier corners of AI evaluation. Reasoning benchmarks like Big Bench Audio tell you if a model can think, and simulated harnesses like tau-Voice tell you if it can call tools, but neither answers the question that matters when a real person picks up a phone: was that actually a good conversation? Artificial Analysis just launched the Speech Agent Arena to close that gap, and the first leaderboard has already produced an uncomfortable finding for anyone shipping voice products.
How the arena actually works
The Speech Agent Arena is a blind preference benchmark that evaluates which native audio model participants prefer in live voice conversations. In each round, a participant receives one scenario, completes it separately with two hidden models, and records a forced overall preference after both calls. Participants also answer diagnostic questions, which we record separately from the overall preference vote.
The Arena includes 35 scenarios: 15 agentic scenarios with tool calls and 20 non-agentic scenarios without tools. Agentic tasks include things like booking a new-patient dental check-up or ordering two pizzas and a side under a $45 budget. Non-agentic tasks are pure information exchange: asking about Sunday pool hours or beginner yoga class pricing.
Two things about the scoring matter. First, Preference Elo is calculated separately across all scenarios, agentic scenarios, and non-agentic scenarios using Bradley-Terry maximum likelihood, with GPT Realtime 1.5 pinned at 1000 as the anchor. Second, task success is judged by two chained LLM judges: one filters out conversations where the human participant deviated from the assignment, the second checks whether the model made the correct final tool call with the right arguments. Supporting calls and end_call do not count as completion.
The leaderboard, and the split brain problem
Here is where it gets interesting. The model humans most enjoyed talking to is not the model that most reliably completed the task.
| Model | Preference Elo | Task Success Rate |
|---|---|---|
| Gemini 3.1 Flash Live Preview - Minimal | 1046 | 74.6% |
| Gemini 3.1 Flash Live Preview - High | 1014 | 71.8% |
| GPT-Realtime-1.5 | 1000 | 85.1% |
| GPT-Realtime-2 (High) | 914 | 89.8% |
| Grok Voice Think Fast 2.0 High | 908 | 94.7% |
| GPT-Realtime-2.1 High | 892 | 91.5% |
| ElevenLabs Agents (Cascaded) | 937 | 90.5% |
Gemini 3.1 Flash Live Preview - Minimal tops the preference chart but completes only three quarters of the assigned tool calls. Grok Voice Think Fast 2.0 High finishes 94.7% of tasks but sits ninth on preference. As the team put it, some conversations can sound as though the requested action was completed even when the required final tool call was unsuccessful. In other words, the model was pleasant, confident, and lying.
Why latency shows up everywhere
The other clear signal is that responsiveness dominates the feel of a voice agent. Preference Elo generally increases as Time to First Audio decreases. Gemini 3.1 Flash Live Preview - Minimal answers in 0.96 seconds, GPT-Realtime-2 High takes 1.14 seconds, and Qwen Audio 3.0 Realtime Plus lags at 1.54 seconds and 699 Elo. TTFA is a critical indicator of perceived responsiveness in voice agent applications. Participants also flagged that highly preferred models sounded more natural and produced fewer weird audio artifacts.
The price picture
- Gemini 3.1 Flash Live Preview - Minimal: $1.50 per hour of input audio, 74.6% task success
- Grok Voice Think Fast 2.0 High: $4.80 per hour, 94.7% task success
- GPT-Realtime-2.1 High: $10.75 per hour, 91.5% task success
That is a roughly 7x price spread between the cheapest preference leader and the most expensive high-reliability option, with Grok sitting in a genuinely interesting middle position on cost per successful task.
What this changes for people building voice products
The arena replaces Conversational Dynamics in the Speech to Speech Index, which now weights Speech Reasoning, tau-Voice agentic performance, Arena Preference, and Task Success Rate equally at 25% each. Practically, that formalizes something teams shipping voice agents have been muttering about for a while:
- Automated benchmarks that use synthetic customers systematically overestimate how good a model feels to a real user.
- Preference and reliability are separate axes, and picking the highest Elo without checking task success can ship a charming agent that quietly fails to book the appointment.
- Cascaded pipelines like ElevenLabs Agents (Scribe v2 Realtime, GPT-4o Mini, Eleven v3) are competitive on both preference and task completion, which means native audio models have not yet won by default.
- For customer-facing flows where a missed tool call is expensive (payments, bookings, refunds), Task Success Rate should be the primary filter and Elo the tiebreaker.
The scenario prompts and tool schemas are kept private except for one worked example on new-patient dental booking, which limits overfitting but also limits reproducibility. Still, this is the first public leaderboard that seriously separates whether a voice agent is enjoyable from whether it actually works, and the two answers are further apart than most people expected.
Comments
