Hume AI's VoiceEQ Benchmark Exposes What 1M Human Ratings Reveal About Voice Models

Voice AI benchmarks have had a dirty secret for a while: they measure the wrong things. Word error rate (WER) tells you if a model transcribed correctly. Latency tells you if it responded fast. Neither tells you whether a hesitant "...yeah" was understood differently from a confident "yeah" -- and in a fraud alert or a medical conversation, that distinction is everything. Hume AI is now putting a number on that gap.

Hume has launched Real World VoiceEQ, a public benchmark that evaluates voice AI on the dimensions humans actually care about: tone, pacing, hesitation, emotional fit, speaker consistency, and conversational appropriateness. It is free to explore on Hugging Face right now.

The biggest dataset of human voice ratings ever assembled

Real World VoiceEQ evaluates more than 40 leading proprietary and open-source voice models across 15+ key evaluation dimensions and more than 60 metrics, spanning ASR, TTS, Speech-to-Speech (S2S), and Speech Understanding. The scale of the human data behind it is what makes this unusual.

The benchmark was developed from more than 1 million individual human ratings collected across different demographics, speaking styles, and acoustic environments. The current release includes 785,000 TTS ratings and 48,000 STS ratings, making it one of the largest human evaluations of voice AI conducted to date.

Every evaluation was conducted using Kairos, Hume's flexible, voice-native evaluation platform. The same infrastructure enables frontier AI labs and enterprises to run custom evaluations tailored to specific use cases, identify granular failure modes in production voice systems, generate human preference data, and continuously improve models through reinforcement learning and human feedback.

What the leaderboard actually measures

The benchmark is split across four major task categories:

• Text-to-Speech (TTS): naturalness, expressiveness, emotional delivery, voice consistency, and instruction-following (e.g., "whisper like you're sharing a secret")
• Speech-to-Speech (S2S): whether a model actually uses the audio signal -- tone, pacing, hesitation -- or just treats it like a transcript
• Speech Understanding: emotion recognition, speaker identification, and paralinguistic awareness (paralinguistics = the non-word information carried in voice)
• ASR Robustness: transcription accuracy under real-world conditions like accents, background noise, and overlapping speakers

Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. VoiceEQ is designed to close that gap.

No model wins everything -- and that is the point

The headline finding from the benchmark is not a ranking. It is a pattern. The race for a single "best" voice model is giving way to a collection of specialized capabilities. In practice, this means the model you pick should depend on what you are building.

Speech-to-Speech models showed the widest variation of any category evaluated. Some systems recognized emotion exceptionally well but struggled to respond naturally. The inverse was also true -- some models sound fluid and natural but miss the emotional context entirely.

A concrete example from the benchmark illustrates this well. When asked to deliver a joke, Gemini 3.1 Flash TTS scored 3.80 on average across humor, punchline intonation, and voice quality. Qwen3-TTS scored 1.56 on the same sample -- with a punchline intonation score of 1.00. The words were identical. The delivery was not.

On the TTS side, today's leading systems optimize for different strengths -- including technical accuracy, emotional understanding, conversational intelligence, expressiveness, and robustness. In Hume's TTS evaluations, no single system configuration ranked in the top five across all eight capability groups.

The listening problem nobody is measuring

Perhaps the sharpest finding is about S2S models specifically. Voice models can sound like different people over the course of a conversation, miss hesitation or uncertainty, and struggle with accents, noise, or emotional speech. But the deeper issue is that many S2S systems are not really using the audio at all.

Some systems remained largely transcript-driven -- relying on the words being spoken while overlooking cues like tone, pacing, hesitation, emphasis, and volume. Having access to audio does not mean a model uses what is in it. Humans naturally use these cues to infer confidence, uncertainty, frustration, sarcasm, and empathy. Most models today do not.

Why automated judges fall short here

One of the more technically interesting findings is about using language models as evaluators -- a common shortcut in model development. The Kairos infrastructure enables frontier AI labs and enterprises to run custom evaluations tailored to specific use cases, identify granular failure modes in production voice systems, and generate human preference data. But Hume found that automated judges have real limits in this domain.

When comparing leading speech-language models (SLMs) with trained human raters, agreement was highest on tasks with clear, verifiable answers like pronunciation accuracy. Agreement broke down on more subjective tasks -- SLMs sometimes inferred emotion from the text rather than the audio, and were weakest at judging whether a voice fit a character role or maintained a consistent identity across a conversation. For those judgments, human listeners remain the ground truth.

There is also a benchmark contamination concern. Unlike purely automated scoring systems, human ratings represent actual human perception, making the benchmark significantly more representative of real-world expectations. Hume found signs that some models may have been optimized against existing public benchmarks -- reproducing known errors in reference transcripts and reconstructing masked words that were not present in the audio.

Who should care about this

If you are building or evaluating voice agents, this benchmark is directly relevant. Practical use cases where VoiceEQ-style evaluation matters:

• Customer support agents: detecting frustration or hesitation in callers, not just transcribing their words
• Healthcare and mental health: where emotional tone carries clinical signal
• Fraud detection: where a hesitant confirmation is meaningfully different from a confident one
• Conversational AI products: where voice consistency and naturalness directly affect user trust
• Model selection: choosing the right TTS or S2S model for your specific task rather than defaulting to a single leaderboard winner

The public leaderboard is live on Hugging Face and free to explore. You can sort the heat-mapped tables, listen to curated audio examples, and download results. Hume also offers custom private evaluations through Kairos for teams that need to evaluate their own models or agents against specific production scenarios. The full technical report is available on the Hume AI blog.

A new baseline for the field

Word error rates continue to fall, latency has reached conversational speeds, and many established benchmarks are approaching saturation -- yet anyone who regularly uses voice AI knows something still feels off. VoiceEQ is an attempt to formalize what "off" actually means, and to give the field a shared language for measuring it.

The assumption that a single leaderboard score can tell you which voice model to use is increasingly hard to defend. The field is fragmenting into specialized capabilities, and the teams that build the best voice products will be the ones who understand exactly which dimension they are optimizing for -- and have the evaluation infrastructure to prove it.