Skip to main content
Back

Zyphra's ZAYA1-8B Beats Models 15x Its Size on Math and Coding

Zyphra's ZAYA1-8B packs frontier math and coding reasoning into under 1B active parameters, trained entirely on AMD hardware with a novel test-time compute method.

  • ZAYA1-8B is an open-weight MoE reasoning model with only 760M active parameters that beats models up to 15x its size on math and coding benchmarks.
  • It was trained entirely on AMD MI300X GPUs with no NVIDIA hardware, the first competitive frontier model to do so end-to-end.
  • Three architectural innovations drive efficiency: Compressed Convolutional Attention (8x KV-cache reduction), an MLP-based router with PID-controller balancing, and learned residual scaling.
  • A new test-time compute method called Markovian RSA enables unbounded reasoning with a fixed context window, pushing ZAYA1-8B past Claude 4.5 Sonnet and GPT-5-High on HMMT'25.
  • Weaknesses are real: agentic tasks, tool calling, and complex multi-step instruction following lag behind same-size competitors like Qwen3.5-4B.
  • Available now free on Hugging Face under Apache 2.0, and as a serverless endpoint on Zyphra Cloud.

ZAYA1-8B is a new open-weight reasoning model from San Francisco startup Zyphra that does something the field hasn't seen before: it matches or beats models ten to fifteen times its size on hard math and coding benchmarks, while activating fewer than one billion parameters per token. That's not a rounding error -- it's the result of a full-stack rethink of architecture, pretraining, and inference.

ZAYA1-8B has 8.4B total and roughly 760M active parameters, and was pretrained, midtrained, and fine-tuned end-to-end on 1,024 AMD Instinct MI300X GPUs with no NVIDIA hardware involved. For three years the unspoken rule of frontier model training has been: if the cluster is not NVIDIA, the model is not competitive. Every leading lab built its frontier runs on Hopper or Blackwell silicon. ZAYA1-8B broke that rule.

The efficiency trick hiding in plain sight

Unlike dense models where all parameters activate for every token, MoE models activate only a subset of experts during inference -- ZAYA1-8B activates only a fraction of its total weights, dramatically reducing compute requirements while still retaining strong reasoning capabilities. Because it is sparse MoE, only ~760M parameters fire per token even though 8.4B sit on disk. That gives it the inference economics of a sub-1B model with the knowledge capacity of an 8B one.

The practical consequence is real: you can run this model on hardware that would normally be far too constrained for a model of this capability level. A conversation or document that would normally require 8 GB of KV cache now requires about 1 GB -- this is what makes ZAYA1-8B viable on hardware that would otherwise be too constrained for a model of this capability level.

Three architectural bets that paid off

ZAYA1-8B is built on Zyphra's proprietary MoE++ architecture. It introduces three changes over standard transformer-based MoE designs:

  • Compressed Convolutional Attention (CCA): Unlike standard attention mechanisms that struggle with memory as context windows grow, CCA performs sequence mixing in a compressed latent space, resulting in an 8x reduction in KV-cache size compared to full multi-head attention, enabling more efficient long-context reasoning.
  • MLP-based router: Most MoE models use a linear router to decide which experts handle a specific token. Zyphra replaced this with a more expressive multi-layer MLP-based design. To maintain stability during training -- a common hurdle for MoEs -- they implemented a bias-balancing scheme inspired by PID controllers from classical control theory.