Qwen Ships Qwen3-VL 8B Locally, Seeing Images and Video in 5 GB
Alibaba's Qwen team shipped GGUF weights for Qwen3-VL-8B-Instruct, letting the multimodal model run locally through llama.cpp, Ollama, and LM Studio.
- Qwen released official GGUF weights for Qwen3-VL-8B-Instruct, enabling local multimodal inference
- Ships in FP16, Q8_0, and Q4_K_M, with a separate FP16/Q8_0 vision encoder (mmproj)
- Q4_K_M build is around 5 GB, runnable on CPU, CUDA, Apple Silicon, and Intel GPUs
- Works out of the box with llama.cpp, Ollama, LM Studio, Jan, vLLM, SGLang, and Docker
- 256K context, 32-language OCR, 2D/3D grounding, GUI agent, and video timestamp localization
- Apache 2.0 license; DocVQA 96.1, MMMU 69.6, MMLU-Pro 68.6% on text benchmarks
Qwen3-VL 8B gets official GGUF weights for local multimodal inference
Qwen has published official GGUF weights for Qwen3-VL-8B-Instruct, its 8-billion-parameter vision-language model. The release lets developers run image, video, and text workloads through common local inference tools without assembling a Transformers environment and custom vision preprocessor.
GGUF packages model tensors and metadata for runtimes such as llama.cpp and Ollama. Qwen separates the language model from the multimodal projector, or mmproj, which contains the vision components needed to convert images and video into representations the language model can process.
Pick the precision per component
| Format | Approximate size | Practical trade-off |
|---|---|---|
Q4_K_M |
5 GB | Mixed 4-bit quantization with the lowest memory requirement |
Q8_0 |
8.7 GB | 8-bit quantization with less compression |
F16 |
16.4 GB | 16-bit floating-point weights with the largest footprint |
| Format | Use |
|---|---|
F16 |
Highest available projector precision |
Q8_0 |
Lower memory and storage use |
The listed language-model sizes exclude the projector, KV cache, image or video buffers, and runtime overhead. A Q4_K_M model paired with a quantized projector can fit within many 16 GB unified-memory systems at moderate context lengths. Memory use rises sharply as context grows, making the full 256K-token window impractical on that hardware.
Separate files allow combinations such as a 4-bit language model with an F16 projector for OCR and grounding tests, or quantized files for both components when memory is constrained. Quantization can change recognition and localization accuracy, so production configurations should be evaluated on representative documents, screenshots, and video.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
