DeepVK's USER2 Beats 7B Models at Russian Search With Just 149M Parameters
DeepVK's new Russian sentence encoder packs 8,192-token context and Matryoshka embeddings into a 149M-parameter model that rivals systems 4x its size.
- DeepVK released USER2-base, a 149M Russian sentence encoder with 8,192-token context.
- Built on RuModernBERT-base with three-stage training: RetroMAE, weakly supervised, then supervised fine-tuning.
- Supports Matryoshka Representation Learning at seven dimensions from 32 to 768.
- Scores 61.12 on MTEB-rus and 54.17 nDCG@10 on MLDR-rus, beating E5-mistral-7b on long docs.
- Uses task prefixes (classification, clustering, search_query, search_document) like Nomic Embed.
- Apache-2.0 licensed, ONNX available, with a 34M small variant also released.
DeepVK has released the USER2-base model, a Russian sentence encoder built on RuModernBERT. It supports inputs up to 8,192 tokens and produces embeddings at seven selectable dimensions. A 34M-parameter USER2-small model targets workloads with tighter memory and latency limits.
At 149M parameters, USER2-base gives developers a compact option for Russian semantic search, retrieval-augmented generation, clustering, and similarity scoring. Its Matryoshka Representation Learning training allows one model to produce useful vectors from 32 to 768 dimensions, reducing vector storage and search costs without training separate encoders.
149M parameters, 8K-token inputs
- Backbone: RuModernBERT-base
- Parameters: 149 million
- Maximum input: 8,192 tokens per text
- Embedding dimensions: 32, 64, 128, 256, 384, 512, and 768
- License: Apache 2.0
- Formats: Safetensors and ONNX
Retrieval drives the benchmark gains
Results published with the model show USER2-base reaching 61.12 on the MTEB-rus mean(task) metric, which averages performance across benchmark task categories. It exceeds the previous USER-base and multilingual-e5-base while approaching encoders with substantially larger parameter counts.
| Model | Parameters | Mean task score |
|---|---|---|
| jina-embeddings-v3 | 572M | 63.45 |
| ru-en-RoSBERTa | 404M | 61.71 |
| USER2-base | 149M | 61.12 |
| multilingual-e5-base |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
