Pronunciation tests reorder the text-to-speech leaderboard

Modern text-to-speech models can produce convincing voices while still mangling email addresses, version numbers, abbreviations, and place names. Artificial Analysis has released a pronunciation benchmark that measures whether models say written text correctly in context, a distinction that matters for developers building voice agents.

Google’s Gemini 3.1 Flash TTS leads overall with 88.1%, followed by xAI TTS at 87.6% and ElevenLabs Eleven v3 at 85.6%. The results also reveal a weak relationship between listener preference and pronunciation reliability: several natural-sounding models rank much lower when reviewers assess specific words and sequences.

A test for what listeners actually hear

Listener-preference arenas measure qualities such as naturalness, pacing, and voice appeal. Word Error Rate typically compares an automated transcript of generated audio with the source text. The new benchmark adds a narrower question: did the model pronounce the highlighted text correctly for its context?

Context can change the expected reading. In one sentence, St. may mean “Saint”; in another, it may mean “Street.” A model must infer the intended form instead of applying one expansion everywhere.

The evaluation uses 454 fixed sentences containing 701 target spans. Each model receives the same text, uses a pinned voice, and runs with its default settings. Artificial Analysis adds no text-normalization layer before synthesis.

Human reviewers hear samples in random order without model names. For each highlighted span, they answer “Yes,” “No,” or “I could not tell.” A model’s score equals its Yes responses divided by its combined Yes and No responses. Votes from reviewers who fail attention checks are discarded.

Because the test fixes the corpus, voice, and default configuration, its scores describe performance under those conditions. Production results can change with another voice, language, model version, or preprocessing pipeline.

Four ways speech models stumble

The benchmark divides difficult text into four categories that expose different failure modes:

• Contextually appropriate: Homographs and abbreviations whose pronunciation depends on the surrounding sentence, such as a wound that is bandaged and a bandage that is wound.
• Expanding shorthand: Numbers, dates, units, and notation such as 6'2", Chapter XVII, and 1 tsp.
• Preserving exact sequences: Codes, file paths, email addresses, and identifiers such as .env.local and a.chen@ucsf.edu.
• Standalone terms: Brand names, place names, and technical terms such as Arkansas, façade, and LaTeX.

Different tasks produce different leaders

Gemini 3.1 Flash TTS records the highest overall score and leads contextual disambiguation at 96.4% and shorthand expansion at 84.4%. xAI TTS leads exact-sequence fidelity at 85.7%, while Alibaba’s Qwen-Audio-3.0-TTS-Plus leads standalone terms at 95.5%.

Category averages show where models struggle most. Contextual appropriateness averages 86.2%, and standalone terms average 86.1%. Shorthand expansion falls to 62.4%, while exact sequences average 62.9%.

Exact-sequence scores range from 5.3% to 85.7%, an 80.4-percentage-point spread. Applications that routinely speak account numbers, URLs, file names, or product versions therefore need category-level testing instead of relying on an overall score.

Naturalness scores miss pronunciation failures

Sonic 3.6 leads the Provider Voice Arena with a 1276 Elo rating, yet it ranks 11th for pronunciation robustness at 74.5%. Gemini 3.1 Flash TTS ranks ninth in the arena with 1201 Elo while taking first place for pronunciation. Breeze TTS 2 has a similar arena rating of 1205 but reaches only 73.6% on the pronunciation test.

These measures capture separate product requirements. An arena rating reflects which sample listeners prefer, while pronunciation robustness measures whether specified text is spoken correctly. Provider selection should include both, along with tests built from the strings an application will generate.

Speed and cost reshape the choice

xAI TTS combines an 87.6% pronunciation score with throughput of 106 characters per second. Realtime TTS-2 Flash reaches 220 characters per second, slightly more than twice the throughput, while scoring 77.8%, which is 9.8 percentage points lower.

Kokoro 82M v1.0 is the fastest model in the cited sample at 242 characters per second and the least expensive at $0.65 per million characters. Its pronunciation score is 55.5%, illustrating the tradeoff that low-cost, high-throughput deployments may face.

Gemini 3.1 Flash TTS leads the benchmark at $18.31 per million characters. Among the listed models scoring above 80%, xAI TTS has the lowest quoted price at $15 per million characters. Eleven v3 costs $100 per million characters, the highest listed price in the comparison.

A deployment checklist for voice agents

Voice agents routinely read names, balances, dates, confirmation codes, URLs, and account numbers under tight latency constraints. A pronunciation error can force repetition, trigger an incorrect action, or send a customer to human support.

1. Test representative strings. Build an evaluation set from real application output, including local names, identifiers, currencies, dates, units, and product terminology.
2. Normalize deterministic formats. Convert predictable notation into speakable text before synthesis when the intended reading is known. The benchmark authors expect normalization to improve shorthand and exact-sequence results substantially.
3. Keep linguistic tests separate. Context-dependent words and standalone terms rely more heavily on the model’s language knowledge, so evaluate those categories directly.
4. Measure the full deployment profile. Compare pronunciation, listener preference, latency, throughput, and price under the same production settings.
5. Retest model updates. Provider changes to voices, normalization defaults, and model versions can alter pronunciation without changing an application’s input text.

Artificial Analysis publishes the full set of comparisons on its TTS models page, including pronunciation-versus-price and pronunciation-versus-throughput charts. Its full methodology explains the scoring and review process.