MIT Gives Transformers a 1.6x Speed Boost With Dynamic Convolutions
Researchers from MIT and MIT-IBM Watson AI Lab show that making convolution filters input-dependent yields a 1.33x–1.60x compute advantage over standard Transformers at scales up to 2B parameters.
Takeaways
- Researchers from MIT and MIT-IBM Watson AI Lab introduce dynamic short convolutions as a new primitive for Transformers, where filters adapt to the input at each position.
- Scaling laws show a 1.33x compute advantage over standard Transformers (QKV placement) and 1.60x when applied after every linear layer.
- Gains are consistent from 150M to 2B parameters and extend to MoE Transformers, Mamba-2, and Gated DeltaNet architectures.
- Custom Triton kernels keep end-to-end training overhead to ~8% for the QKV variant, making the gains practical on H100 GPUs.
- Two parameterizations are offered: low-rank (better accuracy) and head-wise (faster kernel), both adding only ~3% parameters to the base model.
- Limitations: scaling only validated up to 2B dense / 7B MoE; inference not fully optimized; only H100 GPUs benchmarked so far.
Transformers are remarkably good at many things, but they have a blind spot: local context composition. When two phrases share the same prefix but require different local groupings to understand, standard attention has to work hard and expensively to figure that out. A new paper from researchers at MIT and the MIT-IBM Watson AI Lab proposes a surprisingly clean fix: swap out static short convolutions for dynamic ones, where the filter itself changes based on the current input.
The paper, titled "Dynamic Short Convolutions Improve Transformers", introduces dynamic short convolutions as a new neural network primitive that plugs directly into existing Transformer architectures. The result is consistent, scalable improvements across language modeling experiments from 150M to 2B parameters, with compute advantages measured via scaling laws.
The gap that static convolutions can't close
Short convolutions -- tiny, fixed filters that slide over a sequence and mix neighboring token representations -- have made a quiet comeback in recent years. The Primer architecture found them useful, and they are now standard in linear RNNs like Mamba and DeltaNet. But these are all static: the same filter applies at every position, regardless of what the input actually is.
The problem is that language is full of cases where the right local composition rule depends on context. Consider the phrases "the old can opener" and "the old can swim": the word "can" plays a completely different structural role in each, and figuring that out requires a filter that adapts to what it's looking at. Static convolutions have the right locality bias but lack the expressivity to handle this. Attention can handle it in principle, but doing so wastes quadratic compute on a fundamentally local problem.

How dynamic convolutions work
The core idea is straightforward. In a standard (static) short convolution, a fixed weight vector w is applied at every position. In a dynamic convolution, a small learned projection generates a