MIT and NVIDIA's QDA Automates AI Compression, Beating Top Recipes in 64 GPU-Hours
Researchers from MIT and NVIDIA hand quantization recipe design to a coding agent, using a minutes-long KL proxy to steer the search.
Takeaways
- MIT and NVIDIA introduce QDA, where a coding agent designs quantization recipes per model and precision.
- Minutes-long KL proxy against cached top-128 original-model distributions replaces 23 to 86 GPU-hour full evaluations.
- Library extends llm-compressor; batched GPTQ cuts a 256-expert build from ~16 GPU-hours to 0.42.
- Beats strongest public recipes by 0.9, 1.4, 2.2 points at W4A16, W4A8, W3A16 on four reasoning models.
- On 320B GLM-5.3-Flash at NVFP4, recovers 98.3% of FP8 accuracy versus 96.9% for NVIDIA's checkpoint.
- Each search uses 64 GPU-hours and $19 to $75 in Claude Code tokens; checkpoints run on stock vLLM kernels.
Selecting a post-training quantization recipe for a large language model remains model- and precision-specific, while evaluating each candidate can consume tens of GPU-hours. Researchers from MIT and NVIDIA built QDA to automate that search. The system lets a coding agent compose quantization methods, build checkpoints, and rank them with a low-cost proxy based on KL divergence. Across four reasoning models, QDA found recipes that outperformed the strongest public alternatives under a 64 GPU-hour search budget.
Recipes break when models change
Post-training quantization, or PTQ, reduces a trained model’s numerical precision without retraining it from scratch. Methods including GPTQ, AWQ, Hadamard rotations, and learned rounding address different sources of quantization error, so practitioners often combine them. W4A16, for example, stores weights at 4-bit precision while keeping activations at 16 bits.
The best combination changes across architectures and precision targets. At W4A16, ParoQuant performs best on Qwen3.5-9B, AWQ leads on Qwen3.8-27B, and GPTQ leads on DeepSeek-R1-Distill-Llama-8B. The rankings change again with 3-bit weights. NVIDIA’s ModelOpt therefore maintains model-specific recipes, an approach that requires continual updates as new open-weight architectures arrive.
An automated search must control two major costs. A full evaluation grades final answers produced after long reasoning traces, requiring roughly 10,000 generations per checkpoint. One evaluation takes about 23 GPU-hours for DeepSeek-R1-Distill-Llama-8B and 60 to 86 GPU-hours for the tested Qwen models. Candidate construction adds another bottleneck because many PTQ methods live in separate research repositories and require architecture-specific ports.