Sarvam AI's Indic DiarBench Tests Speech Recognition Across all 22 Indian Languages

Speech recognition for Indian languages has a measurement problem. Most existing benchmarks test a single speaker reading clean audio, but real conversations are messy: people interrupt, talk over each other, and switch languages mid-sentence. Indic DiarBench, released by Sarvam AI and AI4Bharat, is the first open benchmark designed to test both transcription accuracy and speaker identification together, across all 22 scheduled languages of India. The paper has been accepted at Interspeech 2026.

Two problems hiding inside one

Modern speech pipelines typically solve two separate tasks. ASR (Automatic Speech Recognition) converts audio to text. Speaker diarization answers the question "who spoke when" by segmenting audio into speaker turns. Historically, these have been benchmarked independently, which creates a blind spot: a system can transcribe words correctly but assign them to the wrong speaker, or produce clean transcripts in single-speaker settings but fall apart when voices overlap.

A conventional ASR benchmark dataset provides transcripts but not speaker attribution, while a diarization benchmark dataset provides speaker turns but not verified transcripts. Neither can tell us whether the two work together on the same audio. Speaker-attributed ASR is the combined task: given a multi-speaker recording, output a transcript that also tags each word or segment with the correct speaker identity.

Why Indian languages make this harder

Indian conversational speech layers on several additional challenges that most existing benchmarks ignore entirely:

• Code-mixing: Speakers routinely switch between an Indic language and English mid-sentence, sometimes mid-word.
• Dialectal variation: A language like Hindi or Bengali spans enormous regional variation in accent and vocabulary.
• Script diversity: The 22 scheduled languages span four language families (Indo-Aryan, Dravidian, Sino-Tibetan, and Austroasiatic) and use multiple distinct scripts.
• Overlap patterns: Indian conversational norms involve frequent backchanneling and interruption, making speaker separation harder.

Datasets such as CALLHOME, DIHARD, VoxConverse, AliMeeting, and NOTSOFAR-1 expanded coverage to more languages and increasingly realistic conversational settings, but Indian languages have lacked a comparable open benchmark for joint ASR and diarization. The closest prior work, DISPLACE '24, covered only 5 Indic languages and had no ASR annotations.

What the dataset contains

Indic DiarBench spans all 22 scheduled languages of India with approximately 108 hours of natural multi-speaker audio, 1,164 total samples, 485 unique meeting speakers from 189 districts, and an average speech overlap ratio of 12.8%. Speakers range from 2 to 9 per session.

The corpus is split across three acoustic conditions, each testing a different real-world scenario:

• Near-field (~53 hours): Recorded using one close-proximity microphone per speaker in virtual meetings, covering all 22 languages.
• Far-field (~27 hours): Recorded using distant microphones, introducing reverberation, background noise, and variable speaker-to-microphone distances, covering the top 8 languages by native-speaker population.
• In-the-wild (~28 hours): Curated from publicly available YouTube videos to capture unconstrained acoustic environments, covering the 10 most widely spoken Indian languages.

485 unique speakers from 189 districts across urban and rural India are present, covering a range of dialects and educational backgrounds. Same-gender sessions were kept deliberately to make speaker identification harder, and warm-up discussions were discarded to keep only natural, spontaneous speech.

A five-stage annotation pipeline

Getting the annotations right was the core engineering challenge. The team built a human-in-the-loop pipeline with five stages:

1. Bootstrap transcription: Initial transcripts generated using multiple independent ASR systems, presented to annotators as editable drafts.
2. Human transcription and speaker attribution: Professional annotators produce time-aligned, speaker-attributed transcriptions. No machine-generated annotation is retained without human validation.
3. Code-mixed transcription: Each session was transcribed in two formats: one entirely in the native Indic script, and one using Roman script for English words and Arabic numerals. Both formats were accepted while calculating WER.
4. Quality control: Dedicated quality checkers (2-3 per language) verify transcription consistency, code-mixing conventions, speaker timestamps, and labels. Overlapping speech segments require multiple review rounds.
5. Expert review: In-house language-specific experts perform final quality checks.

Three metrics, one picture

The benchmark evaluates systems on three complementary metrics. Each captures a different failure mode:

• DER (Diarization Error Rate): Measures how well a system identifies who spoke when. It captures missed speech, false alarms, and speech assigned to the wrong speaker.
• cpWER (concatenated minimum-permutation Word Error Rate): Measures transcription and speaker attribution together. A wrong word and a correctly transcribed word assigned to the wrong speaker both count as errors.
• WDER (Word Diarization Error Rate): Isolates speaker attribution errors by measuring the percentage of aligned words assigned to the wrong speaker.

You can load the dataset and run your own evaluation in a few lines:

from datasets import load_dataset
ds = load_dataset("sarvam/indic-diarbench", "Hindi", split="test")
sample = ds[0]
print(f"Duration: {sample['duration_seconds']:.1f}s")
print(f"Speakers: {sample['num_speakers']}")
print(f"Condition: {sample['dataset_type']}")
for seg in sample['annotated_transcript'][:3]:
    print(f"  [{seg['start_time']:.1f}-{seg['end_time']:.1f}] {seg['speaker_id']}: {seg['transcript']}")

How current systems actually perform

The benchmark comes with baseline results across a range of commercial and open-source systems. The numbers reveal a significant performance gap between Indic-specialized models and general-purpose APIs:

Even the best-performing commercial API (AWS Transcribe) shows a cpWER of 43.7%, meaning nearly half of all words are either wrong or attributed to the wrong speaker. GPT-4o's cpWER of 83.1% suggests that general-purpose multimodal LLMs are not yet viable for this task at scale. The numbers also show that DER and cpWER don't always move together: Gemini 3 Pro has a very high DER (74%) but a more competitive cpWER (58.9%), suggesting it transcribes words reasonably well but struggles badly with speaker segmentation.

What this unlocks

The practical use cases are immediate. Any team building voice agents, meeting transcription tools, call center analytics, or podcast processing for Indian language speakers now has a rigorous way to measure whether their pipeline actually works in realistic conditions. The benchmark is particularly relevant for:

• Teams evaluating which commercial ASR provider to use for multi-speaker Indic audio
• Researchers training or fine-tuning diarization models for low-resource Indic languages
• Product teams building customer support or healthcare voice tools where speaker attribution matters

The benchmark's current limitation is that it is a test set only, not a training corpus. It also does not yet cover all 22 languages in the far-field and in-the-wild conditions, with several lower-resource languages represented only in the near-field subset. But as a measurement tool, it fills a gap that has made it genuinely difficult to know how well any system handles the reality of Indian conversational speech.

The dataset is released under the CC BY 4.0 license and is fully available on Hugging Face. The paper is on arXiv and will be presented at Interspeech 2026.