KAIST's QMamba Fixes the Biggest Flaw Holding Mamba Vision Models Back
KAIST researchers replace Mamba's rigid image scan order with a query-driven cross-attention projector, cutting visual tokens and boosting both accuracy and throughput on vision-language benchmarks.
- Core contribution: KAIST researchers propose a query-based cross-attention projector that compresses image tokens for Mamba multimodal LLMs, eliminating the need for hand-crafted 2D scan orders.
- Key problem solved: Mamba requires sequential 1D input, but images are 2D; prior models needed brittle manual scanning strategies that passed all tokens unfiltered into the LLM.
- How it works: Learned query vectors attend over all image patch features via cross-attention, producing a compressed, fixed-length visual token sequence tailored to the text query.
- Results: Outperforms prior Mamba VLMs (Cobra, VL-Mamba) on VQAv2, GQA, VizWiz, TextVQA, POPE, and MMBench, with improved throughput due to shorter token sequences.
- Venue: Accepted to EMNLP; paper available at arXiv 2606.04719.
- Limitation: Aggressive token compression can hurt fine-grained spatial tasks; no public code repository announced at publication time.
Transformer-based vision-language models have a well-known Achilles' heel: when using the attention mechanism, information from all previous tokens can be passed to the current token, making a forward pass O(n²) in training. Feed a high-resolution image into one of these models and the token count explodes, turning inference into a compute nightmare. Mamba, the state-space model (SSM) alternative, was supposed to fix this. And it largely does -- but plugging images into Mamba introduced a new, subtler problem that nobody had cleanly solved. A team from KAIST, UIUC, and Korea University now has a strong answer.
Their paper, "Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM", introduces a cross-attention-based projector that compresses visual tokens dynamically based on the text query, and in doing so, eliminates the need to hand-engineer how image patches are fed into Mamba's sequential scan. The result is a cleaner architecture that improves both benchmark scores and throughput over prior Mamba multimodal models like Cobra and VL-Mamba.
Why Mamba and vision don't naturally mix
The Transformer's quadratic complexity with input length imposes an unsustainable computational load on LLMs. In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. Mamba promises similar performance and scaling laws as the Transformer whilst being feasible at long sequence lengths -- say, 1 million tokens.
But Mamba processes sequences in a strict causal order, one token at a time. That works beautifully for text. Images, however, are 2D grids with no inherent reading direction. Prior approaches divide 2D images into patches and adopt specific scanning strategies to flatten the images into multiple 1D sequences from different directions. This means someone has to decide: do you scan left-to-right, top-to-bottom, diagonally? To bridge the gap between 2D non-causal image information and the inherent causal modeling capabilities of SSMs, earlier work like VL-Mamba introduced Vision Selective Scan modules with Bidirectional Scanning and Cross Scanning mechanisms. These are hand-crafted heuristics -- and they come with a second problem: they pass
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.