Skip to main content
Back

Artificial Analysis Ranks Every Voice AI Model Into One Score

Artificial Analysis launches the first composite Speech-to-Speech Index, revealing a hard ceiling on agentic voice tasks where every model scores below 53%

Paper, Full Access
  • New composite index: Artificial Analysis launches the Speech-to-Speech Index, combining Speech Reasoning, Conversational Dynamics, and Agentic Performance into one score.
  • GPT-Realtime-2 leads overall at 77.2%, but Grok Voice Think Fast 1.0 (75.7%) tops the hardest dimension -- agentic task completion -- at 52.1%.
  • Agentic performance is the hard wall: Every model scores below 53% on real-world customer service task completion, versus 85% for the best text agent on identical tasks.
  • Speed vs. quality tradeoff is stark: Deepslate Opal responds in 0.44s but scores 62.1%; GPT-Realtime-2 (High) scores 77.2% but takes 2.33s and costs $4.14/hr.
  • Grok's key innovation is background reasoning -- thinking through edge cases concurrently with audio generation, with no added latency.
  • Gemini 3.1 Flash Live (Minimal) is the cheapest at $1.50/hr, while GPT-Realtime-1.5 is the worst value at $11.44/hr for a 72.0% score.

Evaluating voice AI has always been a fragmented exercise: one benchmark tests whether a model knows when to stop talking, another checks if it can count objects from an audio clip, and a third measures whether it can actually resolve a customer's problem end-to-end. Artificial Analysis just collapsed all three into a single number. Their new Speech-to-Speech Index is the first composite metric for native audio models, and the results expose a field that is simultaneously impressive on reasoning and embarrassingly weak on real-world agentic tasks.

What the index actually measures

The index combines three benchmarks, weighted equally, each targeting a distinct capability:

  • Speech Reasoning (Big Bench Audio): 1,000 audio questions adapted from Big Bench Hard, covering formal logic, navigation, object counting, and Boolean truth problems. The model hears the question and speaks the answer -- no text intermediary.
  • Conversational Dynamics (Full Duplex Bench): Tests whether a model handles the messy mechanics of real conversation -- knowing when not to interrupt during a pause, yielding the floor correctly, responding to mid-sentence interruptions, and continuing to speak when the user says "yeah" or "mm-hmm" (backchannels) rather than treating it as a new turn.
  • Agentic Performance (τ-Voice): A benchmark for real-time voice agents on 278 grounded customer-service tasks across retail, airline, and telecom. The model is prompted as a support agent with access to domain tools and a policy document; success means the database ends up in the correct state after the call.

A model must have valid results on all three to appear in the index. That requirement alone filters out many models that have only been tested on one or two dimensions.

The leaderboard, by the numbers

GPT-Realtime-2 (High) leads the index at 77.2%, followed by Grok Voice Think Fast 1.0 at 75.7%, GPT-Realtime-1.5 at 72.0%, and Gemini 3.1 Flash Live Preview (High) at 69.5%. The gap between first and fourth is only 7.7 points -- tight enough that the choice of model depends heavily on which dimension matters most for your use case.

ModelIndex ScoreTTFA (s)Cost/hr Input
GPT-Realtime-2 (High)77.2%2.33s$4.14
Grok Voice Think Fast 1.075.7%1.25s$3.00
GPT-Realtime-1.572.0%0.82s$11.44
Gemini 3.1 Flash Live (High)69.5%2.98s$1.75
Deepslate Opal62.1%0.44s--

The ceiling nobody is talking about

The most striking finding is not who leads -- it is how low every model sits on agentic tasks. Even under clean conditions, voice agents achieve only 31--51% versus 85% for GPT-5 (reasoning) on text -- a 34--54 percentage point gap. The Artificial Analysis index confirms this ceiling is still very much in place: Grok Voice Think Fast 1.0 leads Agentic Performance at 52.1%, ahead of GPT-Realtime-2 (High) at 39.8%, with every model below 53%.

The failure modes are well-documented. Agents fail at reasoning even when transcription succeeds: misapplied policies, skipped tool calls, confirmed actions they had not completed. Transcription errors concentrate at authentication -- when the agent mishears a name or email address, everything downstream fails because it cannot pull the correct account.

Conversational dynamics tell a different story. Speech Reasoning (Big Bench Audio) is tightly clustered at the top, led by Grok Voice Think Fast 1.0 at 97.1%. Frontier models have largely solved the reasoning-in-audio problem. The hard problem is doing useful things with that reasoning in a live, noisy, multi-turn call.

What makes Grok Voice Think Fast different

Grok Voice Think Fast 1.0's lead on agentic performance is not accidental. The core architectural bet xAI made is background reasoning: the model performs reasoning in the background, thinking through challenging queries in real time with no impact on response latency -- because reasoning models traditionally increase response time by generating intermediate thinking tokens before producing an answer, and hiding that computation from the conversational latency budget requires careful architecture work.

The model has been battle-tested in real-world conditions including telephony audio, background noise, heavy accents, and frequent interruptions, and natively supports 25+ languages. It is already in production: Grok Voice powers Starlink's phone sales and support operation, achieving a 20% sales conversion rate and a 70% autonomous resolution rate with no human in the loop, operating across 28 distinct tools spanning hundreds of workflows.

In about eight months, the voice frontier has moved from 30% (OpenAI's gpt-realtime-1.0) to 67% (xAI's grok-voice-think-fast-1.0), crossing the non-reasoning text line and closing most of the way to the reasoning ceiling. The biggest single jump is the most recent one: a +29 percentage point gain driven by xAI's reasoning-enabled audio-native model, following the same pattern seen in text -- adding explicit reasoning unlocks a step change in tool-use reliability.

Speed and cost: the real tradeoffs

The index score alone does not tell you which model to use. Latency and cost cut across the rankings in ways that matter for production deployments.

  • Fastest: Deepslate Opal has the fastest average time to first audio (TTFA) at 0.44 seconds, followed by GPT-Realtime-1.5 at 0.82 seconds and Grok Voice Think Fast 1.0 at 1.25 seconds.
  • Cheapest: Gemini 3.1 Flash Live Preview (Minimal) is the lowest cost model in the index at $1.50 per hour of input audio, then Gemini 3.1 Flash Live Preview (High) at $1.75, Grok Voice Think Fast 1.0 at $3.00, and GPT-Realtime-2 (High) at $4.14.
  • Best quality-per-dollar: Grok Voice Think Fast 1.0 sits at 75.7% for $3.00/hr -- GPT-Realtime-2 (High) scores 1.5 points higher for 38% more cost, and GPT-Realtime-1.5 scores 3.7 points lower for 281% more cost.

How the benchmarks work under the hood

The Big Bench Audio dataset is fully open on Hugging Face. Questions were generated using 23 synthetic voices from top-ranked TTS models, and answers are evaluated by a judge model (currently Claude Sonnet 4.6) that checks whether the spoken response is correct or incorrect. The benchmark has gone through three versions since December 2024, with the latest fixing a grading issue where verbose responses that discussed wrong answers before landing on the right one were being marked incorrect.

The τ-Voice benchmark, published by Sierra, is more demanding. It extends τ-Bench to live voice interactions with complex turn-taking dynamics, simulating the kind of call an agent would actually receive: a user with an accent calling from a noisy coffee shop, on a spotty connection, over a compressed phone line. Artificial Analysis runs the evaluation across seven distinct voice personas -- including speakers with Bengali, Sichuan Mandarin, French, and Maharashtrian accents -- to stress-test robustness across real-world caller diversity.

The Full Duplex Bench covers four conversational behaviors: pause handling, turn-taking, user interruption handling, and backchannel handling. The v1.5 extension added overlapping phenomena such as background speech, side conversations, and user backchannels. Artificial Analysis uses a weighted average across all four metrics for the Conversational Dynamics score.

What this means for builders

The index gives teams a single number to anchor model selection, but the underlying dimension scores are where the real decisions live. A few practical read-outs:

  • If you are building a voice agent for customer support or sales -- anything requiring multi-turn tool use -- the agentic dimension should dominate your evaluation. Grok Voice Think Fast 1.0 leads here by a wide margin, and its $3.00/hr pricing makes it competitive.
  • If you need the most natural-sounding conversational flow -- a voice interface where interrupting gracefully matters more than completing a task -- GPT-Realtime-2 (Minimal) tops Conversational Dynamics at 96.1% and costs less than the High variant.
  • If latency is the constraint, Deepslate Opal's 0.44-second TTFA is in a class of its own, though its 62.1% index score reflects real tradeoffs on the agentic dimension.
  • If cost is the primary filter, Gemini 3.1 Flash Live Preview (Minimal) at $1.50/hr is the floor -- but its 56.6% index score means you are giving up meaningful capability.

The broader signal from the index is that the field has bifurcated. Speech reasoning is essentially a solved problem at the frontier -- scores cluster between 93% and 97% for the top models. Agentic performance is where the next generation of voice AI will be won or lost, and right now, no model has cracked 53%. That gap is the most important number in the entire leaderboard.