Skip to main content
Back

IGSP Beats the C4 Default by 19 Points When Pruning LLaMA

A new paper proves that no single calibration dataset can preserve all LLM capabilities after pruning, and proposes a mixing protocol that beats the C4 default by +18.8 points

Paper, Full Access
  • Calibration data choice for LLM pruning creates an unavoidable trade-off: high-perplexity sources preserve general language (rho=+0.71) but hurt Math and Code (rho=-0.53, -0.59).
  • No single calibration source can preserve all capabilities simultaneously at high sparsity, contradicting prior conclusions based on averaged metrics.
  • A uniform multi-source mix across 15 sources reaches 58.8% total retention on LLaMA-3.1-8B at 60% sparsity, vs. 40.0% for the C4 default (+18.8 points).
  • IGSP (Information-Guided Self-calibration Protocol) automates multi-source construction using only model self-generation, 4-gram deduplication, and perplexity balancing.
  • IGSP outperforms prior self-calibration baselines by +2.4 to +4.8 points without requiring any external capability-aligned corpora.
  • The paper is available at arxiv.org/abs/2606.03328; no public code repository was linked at time of writing.

When you prune a large language model down to 60% sparsity, you are removing 6 out of every 10 weights. The only thing guiding which weights to keep is a tiny unlabeled calibration set, typically 128 sequences pulled from C4, a web-crawl dataset. A new paper from Hu Xu et al. argues that this default choice is not just suboptimal, it is structurally incapable of preserving everything a model knows. The fix, they show, is to stop looking for the perfect single source and start mixing.

The pruning pipeline, briefly

Post-training pruning methods like SparseGPT and Wanda do not retrain the model. Instead, they use a small calibration set to estimate which weights matter most, then zero out the rest in a single pass. The algorithm leverages explicit activation statistics from a small calibration corpus, often as few as 128 sequences of 2k tokens, to ensure output reconstruction with minimal error. The whole process is fast and requires no gradient updates, which is why it has become the go-to approach for compressing models at deployment time.

All state-of-the-art LLM pruning approaches unanimously choose to use a small set of data from the C4 training dataset to calculate their pruning scores. That default has been questioned before. Prior work showed C4 is not universally optimal, and that arithmetic datasets can match or beat pre-training data on math tasks. But the field's conclusion remained: calibration source has only a modest impact on overall accuracy. This paper challenges that conclusion directly.

Averaging hides the damage

The key insight is that aggregated accuracy is a misleading metric. When you average performance across General language, Commonsense reasoning, Code, and Math tasks, a source that destroys coding ability but preserves general language can look fine. The authors decompose post-pruning capability into these four dimensions and measure each separately across 15 calibration sources.