Artificial Analysis' AA-WER Streaming Benchmark Reveals No Voice AI Model Wins Everywhere
Artificial Analysis launches AA-WER Streaming, the first benchmark jointly measuring accuracy and latency for real-time STT models, with Cartesia, ElevenLabs, and Deepgram leading the Pareto frontier across 25 models.
Takeaways
- Artificial Analysis launched AA-WER Streaming, the first benchmark pairing WER accuracy and latency for 25 real-time STT models.
- Cartesia Ink-2 (semantic endpoints) leads on final accuracy at 3.59% WER / 0.21s; ElevenLabs Scribe v2 Realtime is close at 3.64% WER / 0.14s.
- Deepgram Flux is the fastest model at just 0.020s to final transcript, though at a higher 7.36% WER.
- Final transcription is only ~0.7pp more accurate than first partial on average -- most models are already at peak accuracy on the first partial event.
- Streaming STT pricing spans $2–$17/1k minutes; Cartesia Ink-2 external endpoints ($4, 3.66% WER) leads on price-accuracy trade-off.
- No single model leads across all three test datasets (AA-AgentTalk, VoxPopuli, Earnings22), highlighting the importance of domain-specific evaluation.
Picking a streaming speech-to-text model for a voice agent used to mean choosing between two separate leaderboards: one for accuracy, one for speed. Artificial Analysis just collapsed those into one with AA-WER Streaming, a new benchmark that plots Word Error Rate (WER) and transcription latency as a single paired metric, specifically designed for the voice-agent use case. Twenty-five models were evaluated, and the results reveal a surprisingly competitive field -- with no single model winning everywhere.
Why the old benchmarks weren't enough
Streaming STT is fundamentally different from batch transcription. Instead of sending a complete audio file and waiting for a result, a streaming model receives audio in real time, chunk by chunk, and emits transcripts continuously. In a turn-based voice agent architecture -- where the user speaks, STT transcribes, an LLM generates a response, and TTS synthesizes audio -- STT latency is the first step in the response chain, and a slow model adds directly to the time the user waits before hearing a reply.
The fastest models on latency are often the least accurate, and the most accurate models are often the slowest. Understanding where each provider sits on that curve, under real production conditions, is what determines which STT model is the right fit for a specific use case. AA-WER Streaming makes that tradeoff visible with a Pareto frontier: a line connecting the models that offer the best accuracy at each latency budget.
How the benchmark works
Artificial Analysis evaluates STT models on accuracy, speed, and price, benchmarking both offline (batch) and streaming transcription. The streaming benchmark runs on roughly 8 hours of audio across three datasets, weighted 50% AA-AgentTalk / 25% VoxPopuli / 25% Earnings22. AA-AgentTalk is a proprietary held-out set focused on voice-agent speech; VoxPopuli covers European parliamentary proceedings with diverse accents; Earnings22 contains corporate earnings calls with technical vocabulary and overlapping speakers.
Two latency metrics are reported, both measured from the moment a VAD (Voice Activity Detection) model -- specifically Silero VAD -- detects end of speech:
- Time to Final Transcription: latency to the first confirmed, no-longer-changing transcript. More useful when accuracy is the priority.
- Time to First Partial Transcription: latency to the very first transcript event after speech ends, whether confirmed or not. More useful for near-instantaneous applications like yes/no detection or speculative decoding -- a technique where a downstream model starts generating a response before the transcript is fully confirmed.
Audio is streamed in real time in 20ms chunks, or as recommended by the public model config. Network delay is included in all latency measurements, reflecting the real end-user experience. WER is calculated using Levenshtein distance after normalization through OpenAI's Whisper normalizer, extended with custom rules for phone numbers, IDs, numeric ranges, and hyphenation variants.
Who leads -- and where
The headline numbers from the benchmark:
- Highest accuracy (Final): Cartesia Ink-2 with semantic endpoints at 3.59% WER / 0.21s latency, followed by ElevenLabs Scribe v2 Realtime (3.64% WER / 0.14s) and Cartesia Ink-2 with external endpoints (3.66% WER / 0.09s).
- Highest accuracy (First Partial): ElevenLabs Scribe v2 Realtime at 3.65% WER / 0.13s, followed by Cartesia Ink-2 external endpoints (4.33% / 0.07s) and AssemblyAI U3 Realtime Pro (4.46% / 0.47s).
- Fastest overall: Deepgram Flux at 0.020s (Final) and 0.019s (First Partial), both at 7.36% WER.
The Pareto frontier on First Partial runs: Deepgram Flux (7.36%, 0.019s) → Deepgram Nova-3 Realtime (6.69%, 0.057s) → Cartesia Ink-2 external endpoints (4.33%, 0.072s) → ElevenLabs Scribe v2 Realtime (3.65%, 0.132s). These four models define the efficient frontier: anything off this line is either slower or less accurate than a model that costs the same or less.
No model wins everywhere
Dataset-level results tell a more nuanced story. On AA-AgentTalk, ElevenLabs Scribe v2 Realtime leads both final (2.8%) and partial (2.9%) WER. On VoxPopuli, AssemblyAI U3 Realtime Pro leads final at 2.0%, while Google Chirp 3 Streaming leads partial at 2.2%. On Earnings22, Cartesia Ink-2 with semantic endpoints leads both at 5.3%. The right model depends heavily on your audio domain.
One finding that will change how many teams think about their pipeline: final transcription is only about 0.7 percentage points more accurate than first partial on average. Most models are already at their final accuracy by the time the first partial arrives -- ElevenLabs Scribe v2 Realtime sits at 3.6% WER at both checkpoints, and Deepgram Nova-3 and Flux are identical at both. A handful improve meaningfully by waiting for the final, led by Voxtral Mini Transcribe Realtime (9.6% → 5.3%) and OpenAI GPT Realtime Whisper (7.5% → 5.1%).
The price picture
Streaming STT pricing spans $2 to $17 per 1,000 minutes of audio. The benchmark maps accuracy against price as well:
| Model | Price ($/1k min) | WER (Final) | Latency (Final) |
|---|---|---|---|
| Soniox Realtime | $2 | 11.9% | 0.06s |
| Cartesia Ink-2 (external endpoints) | $4 | 3.66% | 0.09s |
| ElevenLabs Scribe v2 Realtime | $6.50 | 3.64% | 0.14s |
| Deepgram Flux | -- | 7.36% | 0.020s |
| AssemblyAI U3 Realtime Pro | -- | 4.46% | 0.47s |
Cartesia Ink-2 (external endpoints) at $4 and ElevenLabs Scribe v2 Realtime at $6.50 lead the price-accuracy trade-off. Cartesia's Ink claims the lowest WER of any streaming STT model, natively handling structured data -- phone numbers, dates, emails, currencies, and UUIDs. Ink-2 is built with native turn detection, with turn start and turn end signaled directly by the model, with no external VAD to integrate or maintain.
What this means for voice agent builders
In voice agents handling names, numbers, addresses, or product codes, WER is a direct proxy for failure rate on structured data. Transcription errors compound: a misheard account number breaks the downstream LLM call, which breaks the TTS response, which breaks the user experience. That's why a joint accuracy-latency benchmark matters more than either metric alone.
The practical decision tree now looks like this:
- Sub-100ms latency required, accuracy secondary: Deepgram Flux (0.02s, 7.4% WER)
- Best accuracy with fast final: Cartesia Ink-2 external endpoints (3.66%, 0.09s) or ElevenLabs Scribe v2 Realtime (3.64%, 0.14s)
- Best price-accuracy balance: Cartesia Ink-2 external endpoints at $4/1k min
- Domain-specific audio (earnings calls, technical vocabulary): Cartesia Ink-2 semantic endpoints
- Parliamentary / accent-heavy audio: AssemblyAI U3 Realtime Pro or Google Chirp 3 Streaming
Latency under load matters too: that 80ms latency at 3am can become 300ms during peak hours, and lab-quality recordings don't have speakerphone echo, car noise, or cellular compression. The benchmark includes network latency in all measurements, which makes the numbers more representative of real production conditions than most vendor-published figures.
The full interactive leaderboard, including per-dataset breakdowns and the Pareto charts, is available at artificialanalysis.ai/speech-to-text/streaming. The methodology page details the normalization pipeline, timing logic, and dataset construction. Both the VoxPopuli-Cleaned-AA and Earnings22-Cleaned-AA datasets are publicly available on Hugging Face for independent replication.