EOS Automatically Finds Optimal Vision Token Cuts in Just 3 GPU-Hours
A multi-institution research team unifies visual token pruning, merging, pooling, and reweighting into one differentiable search space — and lets gradient descent pick the best operator automatically.
- What: Researchers from U. Sydney, ByteDance, SJTU, and UCLA introduce Efficient Operator Search (EOS), a differentiable framework for visual token reduction in multimodal models.
- Core insight: Pruning, merging, pooling, and reweighting are all special cases of one continuous operator space — EOS searches it with gradients instead of hand-tuning.
- What it searches: Simultaneously optimizes which layers reduce tokens, how many to keep, and which operator blend to use, under a hard compute budget constraint.
- Speed: Each search run takes ~3 GPU-hours on a single 8×A100 node, making per-model and per-task search practical.
- Results: Evaluated across 12 multimodal benchmarks (POPE, MME, GQA, AI2D, etc.) at token budgets as low as 16 retained tokens; EOS matches or beats hand-designed baselines especially under aggressive reduction.
- Availability: Paper only — no code or weights released yet; experiments use LLaVA-1.5 as the base model.
Every team building a multimodal model eventually faces the same bottleneck: vision encoders produce hundreds or thousands of tokens per image, and feeding all of them into a large language backbone is expensive. The standard playbook is to pick a token-reduction operator , prune the unimportant ones, merge similar ones, pool spatially, or softly reweight , and hand-tune it. A new paper from researchers at the University of Sydney, ByteDance, Shanghai Jiao Tong University, and UCLA argues that this manual selection process is unnecessary, and introduces a framework called Efficient Operator Search (EOS) that finds the right operator automatically using gradients.
The problem with picking operators by hand
Scaling multimodal foundation models has led to substantial performance gains, but these gains often come with substantial inference cost, as dense visual tokens are repeatedly processed by large language backbones, especially in high-resolution, multi-image, and long-context scenarios. The community has responded with a wave of hand-crafted solutions. Existing work improves the efficiency of foundation models through human-designed token reduction operators, including pruning, merging, pooling, and adaptive reweighting.
The trouble is that each of these operators embodies a different assumption about which tokens matter and how to handle the ones you drop. Pruning discards tokens entirely. Merging fuses similar tokens into a weighted average. Pooling applies a fixed spatial aggregation. Reweighting redistributes attention mass without removing tokens. Researchers pick one, tune it for a specific model and task, and call it done , but there is no principled reason to believe any single operator is universally best. Efficient multimodal foundation models often rely on manually designed token-reduction operators, such as pruning, merging, pooling, and adaptive reweighting. Although these operators appear different, they can be interpreted as distinct regimes of a shared operator space.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
