Alibaba Shrinks Qwen2.5-VL to 1.5GB so Developers Can Run Vision AI Locally
Qwen's compact vision-language model gets a 4-bit AWQ build, cutting memory to run multimodal reasoning, OCR, and agent tasks on modest GPUs.
- Qwen released a 4-bit AWQ quantization of their 3B vision-language model, over 800K downloads.
- DocVQA score of 91.8 vs 93.0 for BF16, minimal document understanding loss.
- MMMU drops from 51.7 to 49.1, biggest hit on multi-discipline reasoning tasks.
- Supports computer use, phone use, and visual agent workflows with bounding-box outputs.
- Handles video over an hour with dynamic FPS sampling and timestamp-aware mRoPE.
- Serves via vLLM or SGLang with OpenAI-compatible endpoints out of the box.
Alibaba’s Qwen team has published an AWQ checkpoint of Qwen2.5-VL-3B-Instruct. Its four reported benchmark scores trail the BF16 model by 1.2 to 2.6 points, while the quantized weights require substantially less storage and memory bandwidth. At the time of writing, Hugging Face displayed more than 800,000 downloads for the checkpoint. That count includes repeat file fetches and provides no measure of unique users or production deployments.
Qwen2.5-VL-3B is the smallest member of a family that also includes 7B and 72B models. The 3B AWQ release targets local document processing, interface grounding, video analysis, and mixed-media retrieval on hardware with tighter memory limits than large multimodal models typically allow.
AWQ trims weight memory
Activation-aware Weight Quantization examines activation statistics to identify weight channels that contribute most to model output. It rescales those channels to reduce quantization error, then stores supported linear weights at 4-bit precision. Activations, the key-value cache, quantization scales, and some model components remain at higher precision.
Four-bit arithmetic puts the lower-bound storage for 3 billion quantized parameters near 1.5 GB, compared with roughly 6 GB for the same parameter count in BF16. The complete checkpoint and its runtime footprint are larger because of metadata, higher-precision tensors, visual inputs, temporary activations, and the key-value cache. A single consumer GPU is feasible for many workloads, but available context, image resolution, video length, batch size, and concurrency determine the actual VRAM requirement.
The multimodal interface remains intact
The quantized release derives from the BF16 checkpoint and uses the same processor, chat template, and generation workflow. Qwen reports support for the following tasks:
- Reading text, charts, icons, graphics, and page layouts inside images.
- Extracting structured data from invoices, forms, tables, and other scanned documents.
- Returning object locations as bounding boxes or points, including coordinates and attributes in JSON.
- Generating computer-use and phone-use actions from screenshots. A host application must validate and execute those actions.
- Analyzing videos longer than one hour and returning timestamps for relevant events, subject to the available context and frame-sampling budget.
Four tests put a price on compression
Qwen evaluated the BF16 and AWQ variants with
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
