---
title: "SparkLLM's Spark-X2.5 Packs 1M-Token Context Into a Tiny 4B Model"
subtitle: "A compact 4B open-source model with hybrid sliding-window attention, native 1M-token context, and agent-focused benchmarks that top comparable small models."
url: "https://alphasignal.ai/news/sparkllm-s-spark-x2-5-packs-1m-token-context-into-a-tiny-4b-model"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-08-24T06:34:34.000Z"
updated: "2026-09-06T09:15:33.192Z"
type: "Model"
topics:
  - "Llms"
  - "Open Source"
subtopics:
  - "Long Context"
  - "Small Models"
  - "Mixture Of Experts"
upvotes: 2113
source_url: "https://huggingface.co/XHToken/Spark-X2.5-4B"
comments: 0
truncated: true
---

# SparkLLM's Spark-X2.5 Packs 1M-Token Context Into a Tiny 4B Model

> A compact 4B open-source model with hybrid sliding-window attention, native 1M-token context, and agent-focused benchmarks that top comparable small models.

The SparkLLM team has released [Spark-X2.5-4B](https://huggingface.co/XHToken/Spark-X2.5-4B) on Hugging Face, a compact language model that pairs a hybrid attention architecture with a native one-million-token context window. Alongside a smaller 1.7B sibling, it targets on-device and agentic workloads where memory footprint and long-context throughput typically collide.

[The model card](https://huggingface.co/XHToken/Spark-X2.5-4B) covers conversation, writing, translation, reasoning, coding, tool use, and agentic workflows. Apache 2.0 licensed and downloadable from Hugging Face, it ships with runtime support for vLLM, SGLang, MLX, Ollama, and LM Studio.

## The attention layout

Spark-X2.5 uses a hybrid attention architecture that combines one full-attention layer with three sliding-window attention layers. Sliding-window attention restricts each token to a fixed local neighborhood rather than the full sequence, dropping memory and compute cost from quadratic to linear in that window. A single full-attention layer interleaved every few sliding layers preserves information routing across the entire context.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Spark-X2.5-4B model card](https://huggingface.co/XHToken/Spark-X2.5-4B)
- [Spark-X2.5 Collection](https://huggingface.co/collections/XHToken/spark-x25)
- [vLLM project](https://github.com/vllm-project/vllm)