Idiap's Abstract Compression Halves Audio Tokens Without Losing Conversation Context
Idiap and Uniphore researchers show that squeezing prior conversational audio into a handful of latent tokens recovers most context-aware ASR gains while slashing prompt length.
- New paper proposes Abstract Compression for conversational context in LLM-based ASR.
- Prior-turn audio is distilled into K learned latent tokens; transcripts kept as text.
- Built on Phi-4-Multimodal with Perceiver-style cross-attention and two-stage training.
- On WoW out-of-domain set, Bias-WER drops from 25.6 to 24.5 vs 23.3 for raw context.
- Benefits saturate around five context turns; K=16 latents work best.
- Accepted to IEEE SLT 2026; evaluated only on Phi-4-Multimodal backbone.
Speech recognizers often process each utterance without the clues established earlier in a conversation, making uncommon names, places, and domain terms easier to miss. A new paper from Idiap Research Institute, EPFL, Uniphore, and collaborators presents Abstract Compression, a method that preserves part of the accuracy gain from conversational context while reducing the number of audio tokens sent to a multimodal large language model.
Why conversation history bloats the cache
Models such as Phi-4-Multimodal convert audio into long sequences of token embeddings and pass them to a text decoder. Adding earlier turns gives the decoder lexical and acoustic context, but every extra token expands the key-value cache, which stores attention state during generation. Longer sequences therefore raise memory use and computational cost.
Naive multi-turn prompting at inference time performed worse than the single-turn baseline. The model began using conversational history effectively after supervised fine-tuning on multi-turn examples, with the largest gains appearing in Bias-WER. This metric calculates word error rate within annotated entity spans, including names and locations.
K tokens replace each past audio stream
Abstract Compression places a learned cross-attention resampler between previous audio turns and the language model. Learned query matrices associated with each relative turn index compress a variable-length sequence of audio embeddings into exactly
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
