NVIDIA’s 100M Nemotron 3 model targets streaming speaker diarization

NVIDIA has released Nemotron 3 Diarization, a 100-million-parameter model that identifies when each person speaks in recorded or live audio. The open-weight checkpoint supports offline and streaming inference, overlapping speech, and as many as eight speakers. NVIDIA also provides a Hugging Face demo.

Voice Arena’s initial Diarization-Bench results ranked the model first among 12 systems and 17 configurations tested on 139 English-language conversations totaling about 22 hours. Nemotron recorded a 14.72% diarization error rate, or DER, compared with 19.3% for the next-ranked configuration, a 23.7% relative reduction. The benchmark scored overlapping speech and used no boundary collar, meaning it allowed no timing tolerance around speaker transitions.

Speaker labels, 80 milliseconds at a time

Speaker diarization assigns each active speaker a label and a sequence of time intervals. Those intervals can be merged with word timestamps from an automatic speech recognition model to produce a speaker-attributed transcript. DER measures the share of speaker time affected by missed speech, false detections, or assignment to the wrong speaker, so lower values indicate better performance.

A streaming diarizer receives small audio chunks with limited surrounding context. It must preserve speaker assignments between chunks, including after pauses and interruptions. If that mapping drifts, one person can appear as speaker_1 in one chunk and speaker_2 in the next, corrupting transcripts and downstream summaries.

Arrival order prevents label drift

Nemotron follows the Sortformer approach, which orders speakers by their first appearance. The first detected voice occupies channel one, the next new voice occupies channel two, and subsequent voices follow the same rule. This stable ordering removes the need to rematch arbitrary output channels for every chunk.

The checkpoint accepts 16 kHz mono audio and converts it into Mel-spectrogram features, a time-frequency representation of the signal, at 10-millisecond intervals. It stacks eight feature frames into 80-millisecond encoder frames, then processes them with a 31-layer Transformer using rotary positional embeddings to retain frame order.

The default output is a [T, 8] floating-point tensor containing an activity probability for every speaker channel at every time step. Independent channels allow multiple speakers to be active in the same frame. Live inference maintains continuity through three memory structures:

• Arrival-Order Speaker Cache: Retains information about speakers from earlier chunks and associates it with their arrival-ordered channels.
• FIFO context queue: Supplies recent frames immediately preceding the current chunk.
• Right-context buffer: Adds a small amount of future audio to resolve transitions, increasing input-buffer latency in exchange for more context.

Latency set at inference time

The same checkpoint exposes four recommended buffer profiles, allowing applications to select a latency and accuracy tradeoff without retraining the model.

Input-buffer latency equals (chunk_len + right_context) × 80 ms. The published figures exclude model execution, network transport, audio capture, and downstream speech recognition.

Gains beyond the headline benchmark

NVIDIA’s model card compares Nemotron with its earlier four-speaker Streaming Sortformer baseline. At the 1.04-second setting, the new model lowers DER across all eight listed aggregate evaluation conditions. Relative reductions range from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM. The reported 41.0% mean gives each dataset equal weight.

The throughput test reports 15,113 times real-time processing at batch size 32 with torch.compile(), up from 2,619 times real time for the previous model. That is about 5.8 times the baseline throughput on one RTX PRO 5000. In the same comparison, DIHARD III DER falls from 19.09% to 12.73%.

Support for eight speakers produces the clearest gains in higher-speaker-count subsets of DIHARD III, CALLHOME-Part2, and NOTSOFAR1. Those conditions expose the fixed four-speaker ceiling of the earlier checkpoint.

Where accuracy can break down

• Anonymous channels: Labels such as speaker_2 track voices within a recording. Assigning names requires meeting metadata or a separate speaker-verification system.
• Eight-speaker limit: Recordings with more than eight participants can produce missed speech or incorrect channel assignments.
• Acoustic sensitivity: Noise, severe reverberation, far-field microphones, domain shifts, and long conversations can increase misses, false alarms, boundary errors, and speaker confusion.
• English benchmark scope: The headline Voice Arena result covers English-language conversations and does not establish equivalent accuracy for other languages.
• Two-speaker regression: On the two-speaker CALLHOME subset, Nemotron records 5.98% DER, compared with 5.68% for NVIDIA’s previous baseline.

Run the high-context profile in NeMo

from nemo.collections.asr.models import SortformerEncLabelModel

diar_model = SortformerEncLabelModel.from_pretrained(
    "nvidia/Nemotron-3-Diarization"
)
diar_model.eval()

diar_model.sortformer_modules.spkcache_len = 264
diar_model.sortformer_modules.fifo_len = 40
diar_model.sortformer_modules.chunk_len = 340
diar_model.sortformer_modules.chunk_right_context = 40
diar_model.sortformer_modules.spkcache_update_period = 300
diar_model._check_streaming_parameters()

predicted_segments = diar_model.diarize(
    audio=["/path/to/conversation.wav"],
    batch_size=1,
)

The chunk and right-context values in this example produce the 30.4-second input buffer. The method returns time-stamped segments formatted as start end speaker_id strings.

NeMo accepts one audio path, a list of paths, NumPy arrays, or a line-delimited JSON manifest. Supported inputs include 16 kHz, single-channel .wav, .flac, .opus, and .mp3 files.

A word-timestamped ASR model such as Parakeet TDT can supply the transcript text. An application can then align each word or segment with Nemotron’s active-speaker intervals to build a speaker-attributed transcript.

The published deployment setup targets Linux and NVIDIA Ampere, Hopper, or Blackwell GPUs. The weights are governed by the OpenMDW License 1.1.

A smaller voice-processing stack

Meeting assistants, call analytics systems, podcast tools, and multi-party voice agents depend on accurate attribution because speaker errors propagate into summaries, action items, and search indexes. Nemotron gives those applications one checkpoint for recorded and live audio, with latency controlled through inference settings and overlap represented directly in the output tensor.

Teams that currently combine voice-activity detection, speaker embeddings, clustering, and channel-matching heuristics can evaluate a single neural diarizer in their place. Argmax has integrated the model into its on-device SDK for mobile speaker-attributed transcription, while NVIDIA lists Baseten and DigitalOcean as deployment partners.