Jina AI's 33M Model Handles 8,192 Tokens on a Laptop
Jina AI's compact 33M-parameter embedding model packs 8192-token context into a tiny footprint, using bidirectional ALiBi instead of positional embeddings.
PRO- jina-embeddings-v2-small-en is a 33M-parameter English embedding model with an 8,192-token context window.
- Uses bidirectional ALiBi in a custom JinaBERT encoder instead of learned positional embeddings, enabling length extrapolation.
- Pretrained on C4, then fine-tuned on 400M+ curated sentence pairs and hard negatives.
- Trained at 512 tokens but extrapolates to 8k+ without retraining, per the technical report.
- Apache-2.0 licensed, works with sentence-transformers via
trust_remote_code=True, 32.7MB F16. - Strong for long-document RAG on modest hardware; English-only, may need fine-tuning for specialized domains.
Long-context retrieval usually forces a tradeoff: chunk documents aggressively and lose semantic coherence, or pay for a heavyweight model that eats VRAM. jina-embeddings-v2-small-en is a 33-million parameter English embedding model that handles sequences up to 8,192 tokens while running comfortably on a laptop.
ALiBi: the trick behind the long context
The model is built on JinaBERT, a BERT variant that replaces learned positional embeddings with ALiBi (Attention with Linear Biases). Instead of embedding position as a learned vector, ALiBi applies a linear bias directly to attention scores. Because those biases generalize beyond token counts seen during training, the model extrapolates well past its training length: it was trained on 512-token sequences yet reliably handles 8,192 tokens. At the time of its release, that represented a 16x increase over the maximum sequence length of leading open-source embedding models.
The bidirectional application of ALiBi is the novel piece here. Prior work used ALiBi exclusively in generative, decoder-only models. Jina's team integrated it into an encoder architecture, making it suitable for producing document embeddings rather than generating text.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
