MCAT Beats SeamlessM4T Across 70 Languages While Shrinking Audio to 30 Tokens

Speech-to-text translation has become one of the flashiest capabilities of modern multimodal LLMs, but under the hood it still has two ugly problems: nearly every training set is centered on English, and the speech encoder tends to spit out huge token sequences that choke inference. A team from Harbin Institute of Technology, Pengcheng Laboratory, Cambridge, and Shanghai Jiao Tong University has released MCAT, a framework that tackles both issues at once, scaling many-to-many translation to 70 languages while compressing 30 seconds of audio down to 30 tokens.

Why English-centric data is a ceiling

Most speech translation benchmarks and training corpora are heavily biased toward English on one side of the pair. This restricts the scaling of MLLMs' many-to-many translation capabilities and pushes models toward being good at X-to-English rather than truly cross-lingual. The second bottleneck is architectural. Current MLLMs often use an adapter similar to LLaVA that projects speech features directly into the LLM through an MLP, producing input sequences of roughly 750 tokens even for very short utterances like "Will it rain tomorrow?" Long sequences mean slow inference, expensive KV caches, and painful batch throughput.

MCAT (Multilingual Cost-effective Accelerated Speech-to-Text Translator) attacks both problems with two contributions: a curriculum learning recipe that expands language coverage to 70 languages with balanced data, and a redesigned speech adapter that shrinks the audio side of the prompt by 25x.

Squeezing 30 seconds into 30 tokens

The pipeline keeps a frozen Whisper-large-v3 encoder for the acoustic front end and then stacks three trainable pieces on top. A Q-Former extracts a fixed-length set of speech embeddings, a temporal pooling layer downsamples the sequence, and an MLP aligns the compressed features to the LLM's embedding dimension. Concretely, Whisper produces a 1500-token representation, the Q-Former reduces this to 150 learned query tokens at dimension 768, and pooling brings that down to just 30 tokens before the MLP hands the vectors to the LLM.