LLMCodec Compresses LLaMA-3-8B to 2-Bit With 36% Better Quality

LLMCodec repurposes battle-tested video compression technology to shrink LLMs at 2-bit precision, beating existing quantization methods by 21% on downstream tasks

·
·
·
LLMCodec Compresses LLaMA-3-8B to 2-Bit With 36% Better QualityPRO
Read2 min
TypePaper
TopicLlms · Gpus
  • LLMCodec compresses LLM weights by treating weight matrices as video frames and encoding them with the VVC/H.266 video codec.
  • On LLaMA-3-8B at 2-bit precision, it reduces perplexity by 36% and improves downstream task accuracy by 21% vs. FlatQuant.
  • A learnable affine transformation is applied before encoding to suppress weight outliers, which is the key innovation over naive codec compression.
  • The method is calibration-free and training-free -- no representative dataset needed, unlike GPTQ or AWQ.
  • VVC/H.266 outperforms JPEG, WebP, and HEVC/H.265 for weight compression; the QP parameter controls the size-quality tradeoff.
  • No public code repository was released; the pipeline depends on the open-source VVenC encoder.

Getting a large language model to fit on a device with limited memory is one of the most practical engineering challenges in AI today. The dominant approach has been quantization -- reducing the numerical precision of a model's weights from 32-bit floats down to 4-bit or even 2-bit integers. But at those extreme low-bit regimes, quality falls off a cliff. A new paper from researchers at City University of Hong Kong and Shanghai Jiao Tong University proposes an unexpected fix: treat your model weights like a video file and compress them with a video codec.

The result is LLMCodec, a compression framework that pipes LLM weight tensors through VVC/H.266 -- the same video standard used to stream 4K content -- and achieves dramatically better quality at 2-bit compression than existing quantization methods.

Why 2-bit is so hard

The increasing scale of LLMs introduces substantial challenges in terms of storage, transmission, and deployment. Running a 7B-parameter model at full 16-bit precision requires roughly 14 GB of memory. Quantizing to 4-bit cuts that to ~3.5 GB with minimal quality loss, but pushing to 2-bit -- which would bring it under 2 GB -- is where things break down.

Scalar methods such as GPTQ and AWQ quantize individual weights using second-order information or activation-aware scaling. These methods perform well at 4-bit but degrade significantly at 2-bit precision, where the discrete grid becomes too coarse to capture per-weight variation. The core problem is outliers: a small fraction of weights with unusually large magnitudes that force the entire quantization grid to stretch, leaving most weights poorly represented.

Existing methods often rely on fine-tuning or calibration data, which exhibit limited generalization across different tensor types. This means you often need a representative dataset just to compress a model, and results can vary unpredictably across different layer types.

The video codec insight

The key observation driving LLMCodec is structural: a weight matrix is just a 2D grid of numbers, and video codecs are extraordinarily good at compressing 2D grids of numbers. Video codecs offer a promising solution for LLM compression, due to their inherent compatibility with matrix-structured data, configurable compression strategies, and the availability of highly optimized, off-the-shelf implementations.

Pro article

This story is for Pro members

You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.

Trending
  • No trending articles

Comments

avatar

Next Reads