--- title: "Artificial Analysis' AA-WER Streaming Benchmark Reveals No Voice AI Model Wins Everywhere" subtitle: "Artificial Analysis launches AA-WER Streaming, the first benchmark jointly measuring accuracy and latency for real-time STT models, with Cartesia, ElevenLabs, and Deepgram leading the Pareto frontier across 25 models." url: "https://alphasignal.ai/news/artificial-analysis-aa-wer-streaming-benchmark-reveals-no-voice-ai-model-wins" author: "AlphaSignal Newsroom" author_url: "https://alphasignal.ai/editorial-team" company: "Artificial Analysis" company_url: "https://alphasignal.ai/company/artificialanlys" published: "2026-05-28T15:34:29.000Z" updated: "2026-06-08T19:42:07.110Z" type: "News" topics: - "Audio" - "Benchmarks" subtopics: - "Speech To Text" - "Realtime Voice" upvotes: 214 read_time: "6 min" source_url: "https://artificialanalysis.ai/speech-to-text/methodology" comments: 0 --- # Artificial Analysis' AA-WER Streaming Benchmark Reveals No Voice AI Model Wins Everywhere > Artificial Analysis launches AA-WER Streaming, the first benchmark jointly measuring accuracy and latency for real-time STT models, with Cartesia, ElevenLabs, and Deepgram leading the Pareto frontier across 25 models. Picking a streaming speech-to-text model for a voice agent used to mean choosing between two separate leaderboards: one for accuracy, one for speed. [Artificial Analysis](https://artificialanalysis.ai/speech-to-text/streaming) just collapsed those into one with **AA-WER Streaming**, a new benchmark that plots Word Error Rate (WER) and transcription latency as a single paired metric, specifically designed for the voice-agent use case. Twenty-five models were evaluated, and the results reveal a surprisingly competitive field -- with no single model winning everywhere. ## Why the old benchmarks weren't enough Streaming STT is fundamentally different from batch transcription. Instead of sending a complete audio file and waiting for a result, a streaming model receives audio in real time, chunk by chunk, and emits transcripts continuously. In a turn-based voice agent architecture -- where the user speaks, STT transcribes, an LLM generates a response, and TTS synthesizes audio -- STT latency is the first step in the response chain, and a slow model adds directly to the time the user waits before hearing a reply. The fastest models on latency are often the least accurate, and the most accurate models are often the slowest. Understanding where each provider sits on that curve, under real production conditions, is what determines which STT model is the right fit for a specific use case. AA-WER Streaming makes that tradeoff visible with a Pareto frontier: a line connecting the models that offer the best accuracy at each latency budget. ## How the benchmark works Artificial Analysis evaluates STT models on accuracy, speed, and price, benchmarking both offline (batch) and streaming transcription. The streaming benchmark runs on roughly 8 hours of audio across three datasets, weighted 50% AA-AgentTalk / 25% VoxPopuli / 25% Earnings22. AA-AgentTalk is a proprietary held-out set focused on voice-agent speech; VoxPopuli covers European parliamentary proceedings with diverse accents; Earnings22 contains corporate earnings calls with technical vocabulary and overlapping speakers. Two latency metrics are reported, both measured from the moment a VAD (Voice Activity Detection) model -- specifically Silero VAD -- detects end of speech: - **Time to Final Transcription:** latency to the first confirmed, no-longer-changing transcript. More useful when accuracy is the priority. - **Time to First Partial Transcription:** latency to the very first transcript event after speech ends, whether confirmed or not. More useful for near-instantaneous applications like yes/no detection or speculative decoding -- a technique where a downstream model starts generating a response before the transcript is fully confirmed. Audio is streamed in real time in 20ms chunks, or as recommended by the public model config. Network delay is included in all latency measurements, reflecting the real end-user experience. WER is calculated using Levenshtein distance after normalization through OpenAI's Whisper normalizer, extended with custom rules for phone numbers, IDs, numeric ranges, and hyphenation variants. ## Who leads -- and where The headline numbers from the benchmark: - **Highest accuracy (Final):** Cartesia Ink-2 with semantic endpoints at 3.59% WER / 0.21s latency, followed by ElevenLabs Scribe v2 Realtime (3.64% WER / 0.14s) and Cartesia Ink-2 with external endpoints (3.66% WER / 0.09s). - **Highest accuracy (First Partial):** ElevenLabs Scribe v2 Realtime at 3.65% WER / 0.13s, followed by Cartesia Ink-2 external endpoints (4.33% / 0.07s) and AssemblyAI U3 Realtime Pro (4.46% / 0.47s). - **Fastest overall:** Deepgram Flux at 0.020s (Final) and 0.019s (First Partial), both at 7.36% WER. The Pareto frontier on First Partial runs: Deepgram Flux (7.36%, 0.019s) → Deepgram Nova-3 Realtime (6.69%, 0.057s) → Cartesia Ink-2 external endpoints (4.33%, 0.072s) → ElevenLabs Scribe v2 Realtime (3.65%, 0.132s). These four models define the efficient frontier: anything off this line is either slower or less accurate than a model that costs the same or less. ## No model wins everywhere Dataset-level results tell a more nuanced story. On AA-AgentTalk, ElevenLabs Scribe v2 Realtime leads both final (2.8%) and partial (2.9%) WER. On VoxPopuli, AssemblyAI U3 Realtime Pro leads final at 2.0%, while Google Chirp 3 Streaming leads partial at 2.2%. On Earnings22, Cartesia Ink-2 with semantic endpoints leads both at 5.3%. The right model depends heavily on your audio domain. One finding that will change how many teams think about their pipeline: final transcription is only about 0.7 percentage points more accurate than first partial on average. Most models are already at their final accuracy by the time the first partial arrives -- ElevenLabs Scribe v2 Realtime sits at 3.6% WER at both checkpoints, and Deepgram Nova-3 and Flux are identical at both. A handful improve meaningfully by waiting for the final, led by Voxtral Mini Transcribe Realtime (9.6% → 5.3%) and OpenAI GPT Realtime Whisper (7.5% → 5.1%). ## The price picture Streaming STT pricing spans $2 to $17 per 1,000 minutes of audio. The benchmark maps accuracy against price as well: | Model | Price ($/1k min) | WER (Final) | Latency (Final) | | --- | --- | --- | --- | | Soniox Realtime | $2 | 11.9% | 0.06s | | Cartesia Ink-2 (external endpoints) | $4 | 3.66% | 0.09s | | ElevenLabs Scribe v2 Realtime | $6.50 | 3.64% | 0.14s | | Deepgram Flux | -- | 7.36% | 0.020s | | AssemblyAI U3 Realtime Pro | -- | 4.46% | 0.47s | Cartesia Ink-2 (external endpoints) at $4 and ElevenLabs Scribe v2 Realtime at $6.50 lead the price-accuracy trade-off. Cartesia's Ink claims the lowest WER of any streaming STT model, natively handling structured data -- phone numbers, dates, emails, currencies, and UUIDs. Ink-2 is built with native turn detection, with turn start and turn end signaled directly by the model, with no external VAD to integrate or maintain. ## What this means for voice agent builders In voice agents handling names, numbers, addresses, or product codes, WER is a direct proxy for failure rate on structured data. Transcription errors compound: a misheard account number breaks the downstream LLM call, which breaks the TTS response, which breaks the user experience. That's why a joint accuracy-latency benchmark matters more than either metric alone. The practical decision tree now looks like this: - **Sub-100ms latency required, accuracy secondary:** Deepgram Flux (0.02s, 7.4% WER) - **Best accuracy with fast final:** Cartesia Ink-2 external endpoints (3.66%, 0.09s) or ElevenLabs Scribe v2 Realtime (3.64%, 0.14s) - **Best price-accuracy balance:** Cartesia Ink-2 external endpoints at $4/1k min - **Domain-specific audio (earnings calls, technical vocabulary):** Cartesia Ink-2 semantic endpoints - **Parliamentary / accent-heavy audio:** AssemblyAI U3 Realtime Pro or Google Chirp 3 Streaming Latency under load matters too: that 80ms latency at 3am can become 300ms during peak hours, and lab-quality recordings don't have speakerphone echo, car noise, or cellular compression. The benchmark includes network latency in all measurements, which makes the numbers more representative of real production conditions than most vendor-published figures. The full interactive leaderboard, including per-dataset breakdowns and the Pareto charts, is available at [artificialanalysis.ai/speech-to-text/streaming](https://artificialanalysis.ai/speech-to-text/streaming). The [methodology page](https://artificialanalysis.ai/speech-to-text/methodology) details the normalization pipeline, timing logic, and dataset construction. Both the [VoxPopuli-Cleaned-AA](https://huggingface.co/datasets/ArtificialAnalysis/VoxPopuli-Cleaned-AA) and Earnings22-Cleaned-AA datasets are publicly available on Hugging Face for independent replication. ## Sources - [X post by @artificialanlys](https://x.com/artificialanlys/status/2060021901234458958) - [AA-WER Streaming Leaderboard | Artificial Analysis](https://artificialanalysis.ai/speech-to-text/streaming) - [Speech to Text Benchmarking Methodology | Artificial Analysis](https://artificialanalysis.ai/speech-to-text/methodology) - [Cartesia Ink-2 Streaming STT](https://www.cartesia.ai/ink/) - [ElevenLabs Scribe v2 Realtime API](https://elevenlabs.io/realtime-speech-to-text-api) - [Best Speech-to-Text Providers in 2026 | Coval](https://www.coval.ai/blog/best-speech-to-text-providers-in-2026-independent-benchmarks-and-how-to-choose)