NVIDIA's Nemotron-3-Embed-1B Beats 8B Rivals on Multilingual Retrieval
NVIDIA's new 1B embedding model beats larger rivals on retrieval benchmarks by distilling an 8B teacher into a compact, multilingual powerhouse
PRO- NVIDIA releases Nemotron-3-Embed-1B-BF16, a 1.14B-parameter multilingual embedding model for retrieval and RAG.
- Scores 72.38 NDCG@10 on RTEB, beating NVIDIA's own previous 1B embedding models by over 10 points.
- Built via two rounds of pruning + distillation from a 3B parent, using an 8B model as the knowledge teacher.
- Covers 34 languages and was trained on 8.5M+ data points across 161 dataset files including code, finance, and medical domains.
- Supports Sentence Transformers, Transformers, and vLLM; requires Transformers ≥5.2.0 and a CUDA GPU for BF16 inference.
- Available now on Hugging Face under the commercial-friendly OpenMDW-1.1 license.
NVIDIA just released Nemotron-3-Embed-1B-BF16, a 1.14B-parameter text embedding model built for multilingual retrieval and RAG pipelines. The headline number is striking: on the RTEB benchmark, it scores 72.38 NDCG@10, outperforming NVIDIA's own previous 1B embedding models by more than 10 points. That kind of gap, at that size, is not incremental.
A small model with a big family tree
The architecture story here is what makes this release interesting. Nemotron-3-Embed-1B-BF16 was derived from the nemotron-3-embed-3b text-embedding model through two iterative rounds of structured pruning and distillation. The process works in stages:
- The 3B parent model was first pruned to 2B using NVIDIA ModelOpt's Neural Architecture Search (NAS), which searches across hidden width, FFN size, attention heads, and depth, then selects the best candidate from the top-10 Pareto front.
- The resulting 2B model was then distilled from the fine-tuned Nemotron-3-Embed-8B-BF16 embedding teacher model, using combined cosine distance loss and mean squared error loss on a multilingual, in-domain retrieval data blend.
- The same pruning-and-distillation procedure was then repeated to produce the final 1.14B embedding model.
The key insight is that the 1B model never learned from scratch. It inherited compressed knowledge from an 8B teacher, which explains why it punches so far above its weight class. The model uses bidirectional attention masking, and the final embedding vector is obtained by applying average pooling to the transformer's token-level representations, encoding each input into a dense 2048-dimensional vector.
The benchmark numbers that matter
The model was evaluated on the Retrieval Embedding Benchmark (RTEB), a benchmark designed to reliably evaluate the retrieval accuracy of embedding models for real-world applications. RTEB (Retrieval Embedding Benchmark) is worth understanding: it measures multilingual retrieval quality across production domains such as legal, finance, code, and medical, combining public datasets with private datasets to be robust to training-data contamination and leaderboard overfitting.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
