LLMCodec Compresses LLaMA-3-8B to 2-Bit With 36% Better Quality
LLMCodec repurposes battle-tested video compression technology to shrink LLMs at 2-bit precision, beating existing quantization methods by 21% on downstream tasks
Takeaways
- LLMCodec compresses LLM weights by treating weight matrices as video frames and encoding them with the VVC/H.266 video codec.
- On LLaMA-3-8B at 2-bit precision, it reduces perplexity by 36% and improves downstream task accuracy by 21% vs. FlatQuant.
- A learnable affine transformation is applied before encoding to suppress weight outliers, which is the key innovation over naive codec compression.
- The method is calibration-free and training-free -- no representative dataset needed, unlike GPTQ or AWQ.
- VVC/H.266 outperforms JPEG, WebP, and HEVC/H.265 for weight compression; the QP parameter controls the size-quality tradeoff.
- No public code repository was released; the pipeline depends on the open-source VVenC encoder.
Getting a large language model to fit on a device with limited memory is one of the most practical engineering challenges in AI today. The dominant approach has been quantization -- reducing the numerical precision of a model's weights from 32-bit floats down to 4-bit or even 2-bit integers. But at those extreme low-bit regimes, quality falls off a cliff. A new paper from researchers at City University of Hong Kong and Shanghai Jiao Tong University proposes an unexpected fix: treat your model weights like a video file and compress them with a video codec.
The result is LLMCodec, a compression framework that pipes LLM weight tensors through VVC/H.266 -- the same video standard used to stream 4K content -- and achieves dramatically better quality at 2-bit compression than existing quantization methods.
Why 2-bit is so hard
The increasing scale of LLMs introduces substantial challenges in terms of storage, transmission, and deployment. Running a 7B-parameter model at full 16-bit precision requires roughly 14 GB of memory. Quantizing to 4-bit cuts that to ~3.5 GB with minimal quality loss, but pushing to 2-bit -- which would bring it under 2 GB -- is where things break down.
Scalar methods such as GPTQ and AWQ quantize individual weights using second-order information or activation-aware scaling. These methods perform well at 4-bit but degrade significantly at 2-bit precision, where the discrete grid becomes too coarse to capture per-weight variation. The core problem is outliers: a small fraction of weights with unusually large magnitudes that force the entire quantization grid to stretch, leaving most weights poorly represented.
Existing methods often rely on fine-tuning or calibration data, which exhibit limited generalization across different tensor types. This means you often need a representative dataset just to compress a model, and results can vary unpredictably across different layer types.
The video codec insight
The key observation driving LLMCodec is structural: a weight matrix is just a 2D grid of numbers, and video codecs are extraordinarily good at compressing 2D grids of numbers. Video codecs offer a promising solution for LLM compression, due to their inherent compatibility with matrix-structured data, configurable compression strategies, and the availability of highly optimized, off-the-shelf implementations.