MI-Pruner Cuts 89% of Image Tokens While Keeping 97% AI Accuracy

MI-Pruner decides which image tokens to drop by measuring how much each one shares with the text prompt. It never reads attention maps, so it can be added to off-the-shelf multimodal models without changing them.

·
·
·
MI-Pruner Cuts 89% of Image Tokens While Keeping 97% AI AccuracyPRO
  • MI-Pruner drops image tokens based on how much each shares with the text prompt (mutual information), with no attention maps needed.
  • On LLaVA1.5-7B it keeps over 97% of full performance while removing 88.9% of visual tokens.
  • On video it averages 93.19% accuracy with only 114 visual tokens, or 5.57% of the original.
  • It is training-free, works with FlashAttention, and runs on Qwen3VL, which [CLS]-based pruners like VisPruner cannot handle.
  • Limits: Qwen3VL baselines were built by the authors, MI beats cosine scoring by only 1-2 points, and outside reviewers flagged under-testing.
  • Code is available on GitHub.

Researchers at KU Leuven and Weill Cornell Medicine have introduced MI-Pruner, a training-free method that removes visual tokens before a multimodal large language model processes them. It scores projected image and text features with mutual information, requiring no decoder attention maps or model retraining. On LLaVA1.5-7B, the authors report preserving more than 97% of the unpruned model’s benchmark performance while removing 88.9% of its visual tokens, leaving 64 of 576.

LLaVA-style models convert each image into hundreds of tokens that enter the language-model decoder alongside the prompt. Those tokens increase prefill computation, attention work and key-value cache memory, even when many image patches contribute little to the answer. Pruning them before decoding can reduce latency and memory use while leaving the base model unchanged.

Attention maps are a fragile dependency

Existing visual-token pruners generally rank image patches through attention scores from either the vision encoder or the language-model decoder. FastV reads decoder attention, while VisionZip and VisPruner use signals from the vision encoder.

Signals and integration requirements of representative pruning methods
Method Ranking signal Integration constraint
FastV LLM decoder attention Requires decoder attention scores
VisionZip Vision-encoder attention Depends on encoder attention outputs
VisPruner Vision-encoder attention Uses a [CLS] summary token
MI-Pruner Projected image and text features Requires no attention matrix

Decoder-side ranking complicates integration with optimized inference kernels such as FlashAttention, which computes attention without materializing the full attention matrix. Encoder-side methods can depend on architecture-specific outputs. The authors note that VisPruner expects a [CLS] token, while recent Qwen vision-language encoders do not provide one.

A score for relevance and repetition

Mutual information measures how much knowledge of one variable reduces uncertainty about another. Pointwise mutual information, or PMI, applies that idea to a specific pair, such as an image patch and a prompt token. MI-Pruner constructs PMI-based scores from embedding similarities in the projector’s output space, where image and text tokens share the language model’s representation format but have not entered the decoder.

Pro article

This story is for Pro members

You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.

Trending
  • No trending articles

Comments

avatar

Next Reads