Skip to main content
Back

Zyphra's Zamba2-7B Beats Llama3-8B While Running 25% Faster

Zyphra's Zamba2-7B-Instruct blends Mamba2 and shared attention to beat Llama3-8B on speed and memory while matching it on quality

  • Zamba2-7B-Instruct is an open-weight 7B chat model from Zyphra combining Mamba2 state-space layers with shared transformer attention blocks.
  • It achieves 25% faster time-to-first-token and 20% higher token throughput than Llama3-8B with a smaller memory footprint.
  • The architecture uses two shared attention blocks in an ABAB pattern plus LoRA adapters, keeping parameter count low while preserving expressivity.
  • Trained on 3 trillion tokens with a dedicated annealing phase; instruct version fine-tuned on chat and instruction-following data.
  • Long-context mode extends to 16k tokens via RoPE frequency adjustment, with reliable retrieval up to ~18k.
  • Released under Apache 2.0, available on Hugging Face and GitHub; requires mamba-ssm kernels for full performance.

Zamba2-7B-Instruct is Zyphra's instruction-tuned small language model that challenges a core assumption baked into most 7B deployments: that you have to pick between quality and inference efficiency. By fusing Mamba2 state-space layers with a small number of shared transformer attention blocks, it manages to beat Llama3-8B on speed and memory while staying competitive on benchmarks.

The KV-cache problem nobody talks about

Standard transformer models pay a hidden tax at inference time: the KV-cache. Every token you generate requires storing key-value pairs for every attention layer, and that memory grows linearly with sequence length and batch size. At 7B parameters, this becomes a real constraint on how many concurrent users you can serve or how long a context you can afford.

The hybrid architecture trades off between the expressivity of full attention and the inference efficiency of SSM blocks. The Mamba2 layers carry the bulk of the computation in linear time and constant state, while the shared attention layers preserve the in-context retrieval capability that pure-SSM models give up. The result is a model that behaves like a transformer from the outside but runs more like a recurrent network under the hood.

What makes the architecture tick

The Zamba2 design is not a naive interleaving of Mamba and attention layers. Several deliberate choices separate it from earlier hybrids:

  • Shared attention weights: The core Zamba architecture consists of a backbone of Mamba2 layers interleaved with one or more shared attention layers. This attention has shared weights to minimize the parameter cost of the model.
  • Two alternating attention blocks: Instead of a single shared attention block, Zamba2 utilizes two shared attention blocks which are interleaved in an ABAB pattern throughout the network.