Jina AI's Open-Source Embeddings Model Reads Whole Documents at 8,192 Tokens
Jina AI's open source English embedding model packs a 137M parameter BERT into a 8192-token context using bidirectional ALiBi.
PRO- Jina v2 base English is a 137M parameter BERT embedding model with an 8192-token context window
- Uses bidirectional ALiBi to extrapolate from 512-token training to 16x longer inputs
- Pretrained on C4, fine tuned on 400M+ curated sentence pairs and hard negatives
- Apache-2.0 licensed, works with sentence-transformers via
trust_remote_code=True - Competitive on MTEB and pairs well with CohereRerank or bge-reranker-large for RAG
- See the technical report and the model card
Long-context embeddings have become genuinely useful infrastructure for retrieval-augmented generation, and jina-embeddings-v2-base-en is one of the few open-source options that pushes the sequence limit far beyond the usual 512 tokens. The model swaps standard positional encodings for a trick borrowed from long-context language models, letting it embed entire documents into a single vector without chunking.
A BERT that reads whole documents
JinaBERT is a 137M parameter English encoder built on a modified BERT architecture that replaces learned position embeddings with ALiBi (Attention with Linear Biases). ALiBi adds a fixed linear penalty to attention scores based on token distance, rather than learning position representations during training. Jina's contribution was adapting it bidirectionally so the technique works in an encoder, where tokens attend to context in both directions simultaneously. The result is a model that supports sequences up to 8,192 tokens, a 16x increase over standard BERT-based embedding models capped at 512.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
