Dynatrace Turns Mamba2 Into a Memory-Efficient Embedder That Beats Transformers
A vertical chunking trick on Mamba2 keeps embedding memory flat regardless of document length, matching transformer quality on standard benchmarks.
PRO- Mamba2 fine-tuned into a text embedder with a vertical chunking inference schedule that keeps memory constant beyond the chunk size.
- Matches transformer quality on MTEB benchmarks, leading on multilingual tasks and tying on LongEmbed.
- Beats Mistral 7B on runtime and memory for single-sequence inputs past 4,096 tokens on an H100.
- Method generalizes to RWKV and xLSTM, any recurrent model with a parallelizable dual formulation.
- English retrieval remains the main quality gap, likely due to weaker pretraining of Mamba2 base models.
- Weights and code released at dynatrace-oss/embed-mamba2 on Hugging Face.
Text embeddings power search, retrieval, and clustering across modern AI systems, but the transformer models that generate them get expensive fast on long inputs. A new paper from Dynatrace Research and TU Wien argues that recurrent architectures can take over that job, offering roughly the same quality at a fraction of the memory cost when documents get long.
The authors fine-tune Mamba2-based language models into general-purpose embedders and introduce an inference schedule that keeps GPU memory constant in the input length once sequences pass a chosen chunk size. They also verify the trick generalizes to two other non-transformer families, RWKV and xLSTM.
Why transformer embedders hit a wall
Transformer-based embedding models suffer from quadratic computational and linear memory complexity, which caps their utility on long sequences. Doubling document length roughly quadruples compute and doubles the activation memory needed to encode the input. For retrieval over 32K-token contracts, papers, or codebases, this quickly exceeds what fits on a single accelerator.
Recurrent models like Mamba2, RWKV, and xLSTM already solve the scaling problem for text generation by maintaining a fixed-size latent state. Their behavior as general-purpose embedders, where you encode a whole input in one shot rather than decoding token by token, has been largely unexplored outside a few task-specific efforts.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
