Skip to main content
Back

NVIDIA's Nemotron 3.5 ASR Streams 40 Languages at 17x More Concurrent Users

NVIDIA released a 600M cache-aware streaming ASR model covering 40 language-locales, and it now ships as GGUF quants that run on a laptop.

  • NVIDIA Nemotron 3.5 ASR is a 600M cache-aware FastConformer-RNNT covering 40 language-locales.
  • GGUF quantizations from 473 MB to 2.38 GB run via transcribe.cpp locally.
  • Runtime-configurable chunk sizes: 80ms, 160ms, 320ms, 560ms, 1.12s, no retraining.
  • 17x more concurrent streams than Parakeet RNNT 1.1B at 80ms on H100.
  • Language-ID prompt conditioning plus auto-detect mode that emits language tags in output.
  • Q8_0 quant hits 7.88% WER on FLEURS English, matching NeMo reference.

A community port of NVIDIA's new multilingual streaming speech recognition model just crossed 1.9 million downloads on Hugging Face, packaged as GGUF quantizations that run through a llama.cpp-style transcribe binary. The underlying model, nvidia/nemotron-3.5-asr-streaming-0.6b, is a 600M parameter cache-aware FastConformer-RNNT that transcribes 40 language-locales from a single checkpoint while streaming audio in chunks as small as 80ms.

What makes this release notable is how much it folds into one small package: low-latency streaming, wide language coverage, punctuation, capitalization, and automatic language detection in a model the size of a Whisper-tiny download. NVIDIA reports the architecture serves 17x more concurrent streams than the previous multilingual baseline.

What actually shipped

Nemotron 3.5 ASR extends nemotron-speech-streaming-en-0.6b by adding language-ID prompt conditioning, supporting transcription across 40 language-locales from a single 600M parameter checkpoint with native punctuation and capitalization, plus runtime-configurable chunk sizes of 80ms, 160ms, 320ms, 560ms, and 1120ms. The GGUF release from handy-computer wraps that checkpoint for transcribe.cpp, a C++ inference runtime in the spirit of whisper.cpp.

The quantization ladder is where this becomes practical for local use:

  • F32 at 2.38 GB, 7.97% WER on FLEURS English test
  • F16 at 1.19 GB, 7.97% WER
  • Q8_0 at 716 MB, 7.88% WER
  • Q6_K at 593 MB, 8.02% WER
  • Q5_K_M at 534 MB, 8.15% WER
  • Q4_K_M at 473 MB, 8.49% WER

The reference NeMo baseline on the same manifest is 7.99%, so Q8_0 matches or slightly beats the full-precision reference within noise. On LibriSpeech test-clean the F32 preset lands at 3.04% against a NeMo reference of 3.03%. Quantization costs almost nothing here until you drop to 4 bits.

Why cache-aware streaming changes the math

Traditional streaming ASR uses buffered inference: every few hundred milliseconds you re-feed a sliding window of audio through the encoder, which means recomputing frames you already processed. Cache-aware streaming, described in the