Skip to main content
Back

QIG Squeezes Vision AI Models to 3 Bits With 1.60% Accuracy Gain

A new post-training quantization method scores every token individually using integrated gradients, closing the gap to full precision to just 1.33 percent at 3-bit weights.

Pro

  • QIG scores each token's quantization sensitivity using integrated gradients, replacing coarse modality-level weighting
  • Improves LLaVA-onevision-7B by 1.60% at 3-bit weights, closing FP16 gap to just 1.33%
  • Adds only 2 minutes to calibration versus MBQ, unlike expensive leave-one-out perturbation methods
  • Beats MBQ, GPTQ, AWQ, SmoothQuant across LLaVA, InternVL2 (8B/26B), Qwen2-VL under W3A16 and W4A8
  • Also improves pure LLM quantization when combined with GPTQ on LLaMA-2-7B at 3-bit
  • Open-source code available at github.com/ucas-xiang/QIG as plug-in for existing PTQ pipelines

Squeezing large vision language models down to 3 or 4 bits is one of the cheapest ways to make them deployable, but the existing playbook has a blind spot. Most current methods treat every image token the same and every text token the same, then try to balance those two buckets. A new paper argues that this is far too coarse, and introduces Quantization-aware Integrated Gradients (QIG), a way to score how sensitive each individual token is to quantization noise and use those scores to guide calibration.

Where modality-level quantization leaks accuracy

Post-training quantization (PTQ) works by minimizing the reconstruction error of each transformer block on a small calibration set, without any retraining. For vision language models, recent work like MBQ (Modality-Balanced Quantization) reweights those errors so that visual and textual tokens do not drown each other out. It helps, but the paper shows the assumption behind it is wrong.

Activation distribution heatmaps across two transformer layers showing massive outliers, layer heterogeneity, sub-layer divergence, and token variability

Looking at activation distributions in InternVL2-8B, the authors identify four recurring phenomena: massive activations, layer heterogeneity, sub-layer divergence, and token variability. Within the same modality, individual tokens have wildly different activation magnitudes and therefore wildly different sensitivity to being rounded to 3 or 4 bits. Treating all vision tokens as equally sensitive throws away that structure.