Sarvam AI's Indic DiarBench Tests Speech Recognition Across all 22 Indian Languages
Sarvam AI releases the first open benchmark for multi-speaker speech recognition across all 22 scheduled Indian languages, exposing major gaps in commercial APIs
- First-of-its-kind benchmark: Indic DiarBench is the first open dataset for joint speaker diarization and ASR across all 22 scheduled Indian languages.
- Scale: ~108 hours of natural multi-speaker audio, 1,164 samples, 485+ unique speakers from 189 districts across India.
- Three acoustic conditions: Near-field meetings, far-field recordings, and in-the-wild YouTube audio, capturing real-world diversity.
- Commercial APIs fall short: Best commercial API (AWS Transcribe) achieves 43.7% cpWER; GPT-4o reaches 83.1%, showing major gaps remain.
- Three evaluation metrics: DER, cpWER, and WDER together measure whether systems get words right and assign them to the correct speaker.
- Fully open: Dataset on Hugging Face (CC BY 4.0); paper accepted at Interspeech 2026.
Speech recognition for Indian languages has a measurement problem. Most existing benchmarks test a single speaker reading clean audio, but real conversations are messy: people interrupt, talk over each other, and switch languages mid-sentence. Indic DiarBench, released by Sarvam AI and AI4Bharat, is the first open benchmark designed to test both transcription accuracy and speaker identification together, across all 22 scheduled languages of India. The paper has been accepted at Interspeech 2026.
Two problems hiding inside one
Modern speech pipelines typically solve two separate tasks. ASR (Automatic Speech Recognition) converts audio to text. Speaker diarization answers the question "who spoke when" by segmenting audio into speaker turns. Historically, these have been benchmarked independently, which creates a blind spot: a system can transcribe words correctly but assign them to the wrong speaker, or produce clean transcripts in single-speaker settings but fall apart when voices overlap.
A conventional ASR benchmark dataset provides transcripts but not speaker attribution, while a diarization benchmark dataset provides speaker turns but not verified transcripts. Neither can tell us whether the two work together on the same audio. Speaker-attributed ASR is the combined task: given a multi-speaker recording, output a transcript that also tags each word or segment with the correct speaker identity.
Why Indian languages make this harder
Indian conversational speech layers on several additional challenges that most existing benchmarks ignore entirely:
- Code-mixing: Speakers routinely switch between an Indic language and English mid-sentence, sometimes mid-word.
- Dialectal variation: A language like Hindi or Bengali spans enormous regional variation in accent and vocabulary.
- Script diversity: The 22 scheduled languages span four language families (Indo-Aryan, Dravidian, Sino-Tibetan, and Austroasiatic) and use multiple distinct scripts.
- Overlap patterns: Indian conversational norms involve frequent backchanneling and interruption, making speaker separation harder.