Artificial Analysis Fixes the Benchmark That Let Bad TTS Models Win

TTS benchmarks have always had a dirty secret: a model that ships beautiful default voices looks great on a leaderboard, even if its underlying synthesis engine is mediocre. Artificial Analysis just closed that loophole with the launch of its Controlled Voice Arena Leaderboard, a new evaluation that clones the exact same 8 voices across every model and then lets human listeners vote blind.

The variable that was skewing every benchmark

The existing Provider Voice Arena lets each model use its own curated voices. That is fine for comparing finished products, but it conflates two very different things: how good a model's synthesis engine is, versus how good its voice designers are. If ElevenLabs ships a perfectly tuned US female voice and a competitor ships a mediocre one, the competitor loses even if its underlying model is technically superior.

The Controlled Voice Arena standardizes the comparison by cloning the same 1-2 minute reference recordings across every model. Each model is evaluated on the same 8 voice categories: 2 US Male, 2 US Female, 2 UK Male, and 2 UK Female voices. The result is a leaderboard that measures synthesis quality in isolation, not voice curation skill.

How the scoring works

The evaluation uses blind human preference testing. Listeners are presented with pairs of speech clips generated from identical prompts, without knowing which provider produced which clip, and simply select the one they prefer. This eliminates brand bias and ensures rankings reflect the actual listening experience. Those preference judgments are aggregated using an Elo rating system, the same framework used in competitive chess and LMSYS Chatbot Arena for evaluating large language models.

Who came out on top

The overall rankings under controlled conditions:

• Overall #1: Cartesia Sonic 3.5 (Elo 1,122)
• Overall #2: ElevenLabs Eleven v3 (Elo 1,088)
• Overall #3: Inworld Realtime TTS-2 Research Preview (Elo 1,070)
• US accent #1: Cartesia Sonic 3.5 (1,139), followed by ElevenLabs Eleven v3 (1,104)
• UK accent #1: Cartesia Sonic 3.5 (1,103), with Inworld TTS-2 (1,075) jumping ahead of ElevenLabs (1,067)
• Open weights #1: Fish Audio S2 Pro (1,034), followed by Mistral Voxtral TTS (1,024) and Resemble AI Chatterbox (930)

One notable shift: Inworld TTS-2 performs better on UK accents than US, moving from third overall to second in the UK category. ElevenLabs, conversely, is stronger on US voices. These are the kinds of nuances that get buried when each model picks its own showcase voices.

The models worth knowing

Cartesia's Sonic 3.5 is designed for low-latency, conversational AI applications, with sub-90ms latency and native support for 42 languages. It follows transcripts faithfully and stays expressive enough to carry a real conversation. A defining technical feature is its State Space Model (SSM) architecture. While most modern foundation models use transformers, which scale quadratically with sequence length, Sonic's SSM scales linearly, enabling higher throughput and lower computational costs at scale.

Inworld's Realtime TTS-2 takes a different approach. The model hears the full audio of the exchange, picks up the user's tone, pacing, and emotional state, then takes voice direction in plain English the way developers prompt an LLM. The meaningful architectural distinction is that it operates as a closed-loop system, taking actual audio of prior turns as input rather than just a transcript.

Inworld's natural-language steering lets developers direct any voice with bracketed cues like [say excitedly], [whisper in a hushed style], or [say sadly with deliberate pauses in a low voice]. There is no preset list; you write directions in natural language.

The open-weights story

The open-weights tier is where things get interesting for teams that want to self-host or avoid vendor lock-in. The gap between open-source and commercial TTS has nearly closed. In 2023, the best open-weight model trailed commercial leaders by 223 Elo points on the Artificial Analysis Speech Arena. By mid-2026, that gap is down to 81 Elo points, and in blind tests, individual open-source models now beat ElevenLabs more often than not.

Fish Audio S2 Pro, the open-weights leader in this controlled evaluation, is trained on over 10 million hours of audio data covering more than 80 languages. It combines a Dual-Autoregressive (Dual-AR) architecture with reinforcement learning alignment to generate speech that is exceptionally natural and emotionally rich. The Dual-AR architecture is a master-slave setup: a large 4B-parameter model handles timing and prosody, while a fast 400M-parameter model fills in the acoustic detail. The model supports fine-grained inline control of prosody and emotion using natural-language tags like [laugh], [whispers], and [super happy], trained on over 10 million hours of audio across approximately 50 languages with reinforcement learning alignment.

One practical caveat: Fish Audio S2 is source-available rather than fully open-source. The model weights are free to use for personal and research purposes, but commercial use has restrictions. Check the license before shipping to production.

What this changes for developers

The practical implication is that you now have two complementary signals when picking a TTS provider. The Provider Voice Arena tells you what a model sounds like with its best foot forward. The Controlled Voice Arena tells you what the underlying engine can actually do when the playing field is level. For teams building voice agents where they will be cloning a brand voice, the controlled leaderboard is the more relevant number.

The arena is live and accepts public votes at artificialanalysis.ai/text-to-speech/arena. The more votes it collects, the more statistically reliable the rankings become. The methodology uses a linear regression model similar to how LMSYS calculates Elo scores for Chatbot Arena, and benchmarking is conducted 4 times daily at random times, with a unique prompt of approximately 500 characters used for each generation.