SparkLLM's Spark-X2.5 Packs 1M-Token Context Into a Tiny 4B Model
A compact 4B open-source model with hybrid sliding-window attention, native 1M-token context, and agent-focused benchmarks that top comparable small models.
PRO- Spark-X2.5-4B released on Hugging Face under Apache 2.0, with a 1.7B sibling.
- Hybrid attention: one full-attention layer per three sliding-window layers, native 1M-token context.
- Pretrained on 20T tokens, post-trained via SFT plus large-scale RL consolidated with MOPD.
- Leads size class on agent benchmarks (MCP-Atlas 54.6, BrowseComp 40.9) and SWE-Bench Pro (44.4).
- Trails larger Qwen3.5-9B and Gemma4-12B on knowledge tasks like GPQA and HLE.
- Runs on vLLM, SGLang, MLX, Ollama, LM Studio; trained on Huawei Ascend clusters.
The SparkLLM team has released Spark-X2.5-4B on Hugging Face, a compact language model that pairs a hybrid attention architecture with a native one-million-token context window. Alongside a smaller 1.7B sibling, it targets on-device and agentic workloads where memory footprint and long-context throughput typically collide.
The model card covers conversation, writing, translation, reasoning, coding, tool use, and agentic workflows. Apache 2.0 licensed and downloadable from Hugging Face, it ships with runtime support for vLLM, SGLang, MLX, Ollama, and LM Studio.
The attention layout
Spark-X2.5 uses a hybrid attention architecture that combines one full-attention layer with three sliding-window attention layers. Sliding-window attention restricts each token to a fixed local neighborhood rather than the full sequence, dropping memory and compute cost from quadratic to linear in that window. A single full-attention layer interleaved every few sliding layers preserves information routing across the entire context.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
