Alibaba Squeezes Qwen3.5-0.8B's Full Multimodal Stack Under 1B Parameters
Alibaba's smallest Qwen3.5 base model packs native vision, 262K context, and a Gated DeltaNet hybrid stack into under a billion parameters.
- Qwen released Qwen3.5-0.8B-Base, a sub-billion-parameter natively multimodal pretrained model under Apache 2.0.
- Architecture uses Gated DeltaNet linear attention in 75% of layers, full attention in 25%, cutting KV-cache memory.
- Native context window of 262,144 tokens, extensible past one million with extrapolation.
- Handles text, image, and video inputs through a single early-fusion backbone, not bolted-on adapters.
- Compatible with Transformers, vLLM, SGLang, llama.cpp quantizations, and Docker Model Runner out of the box.
- Part of a 9-model family spanning 0.8B to a 397B MoE flagship, all sharing the same hybrid stack.
Alibaba’s Qwen team has released Qwen3.5-0.8B-Base, the smallest checkpoint in a family that reaches 397 billion parameters. The 0.8B model carries the family’s native image, video, and text support, along with a hybrid attention stack designed to reduce inference memory during long sequences.
The Apache 2.0 checkpoint is a pretrained base model with no instruction tuning. Raw generations may follow prompts inconsistently, so practical deployments will usually require supervised fine-tuning, preference tuning, or a community instruction-tuned derivative.
| Component | Specification |
|---|---|
| Model type | Pretrained multimodal base checkpoint |
| License | Apache 2.0 |
| Language stack | 24 layers with a 1,024-dimensional hidden state |
| Attention layout | 18 Gated DeltaNet layers and 6 full-attention layers |
| Vocabulary | 248,320 tokens |
| Native context | 262,144 tokens |
| Extended context | More than one million tokens through extrapolation |
| Inputs | Text, images, and video |
Full stack, 0.8B parameters
Native multimodal support means the checkpoint includes the visual processing path used during training. A visual encoder converts images and video into representations consumed by the language stack, giving developers one model interface for media and text tasks.
The 24-layer causal language model repeats the same four-layer pattern six times: three Gated DeltaNet blocks followed by one conventional attention block. Each block also contains a feed-forward network. Multi-token prediction adds training objectives for several future positions, providing a denser learning signal than next-token prediction alone.
The small parameter count reduces weight memory and makes adapter training more accessible. Actual hardware requirements still depend on numerical precision, sequence length, media resolution, optimizer state, batch size, and whether optimized DeltaNet kernels are available.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
