ViT-AdaLA Converts Frozen Vision Models to Linear Attention Within 1% Accuracy
A three-stage recipe converts pretrained softmax Vision Transformers into linear-attention models while staying within 1% of the original accuracy.
ProTakeaways
- ViT-AdaLA converts pretrained softmax ViTs into linear-attention models in three stages: paper link.
- Stage 1 tunes only Q, K, V projections to match softmax attention block by block.
- Stage 2 aligns final-layer features to a frozen VFM teacher via MSE loss.
- Accuracy stays within 1% of softmax backbones on ImageNet-1K with linear complexity.
- Beats Hedgehog, LoLCATS, Performer, Nystromformer, and Monarch on DINOv2 and CLIP.
- Segmentation on ADE20K and Cityscapes shows larger gaps but strong efficiency at high resolution.
Vision Transformers power most modern vision foundation models, but their softmax attention still scales quadratically with sequence length. That is a hard ceiling when you want to run DINOv2 or CLIP on high-resolution imagery, dense segmentation maps, or long video clips. A new arXiv paper called ViT-AdaLA proposes a training recipe that converts an already-pretrained softmax ViT into a linear-attention model, keeping accuracy within roughly one point of the original while removing the quadratic cost.
Why linearizing ViTs has been stubbornly hard
Two families of methods have tried to fix the quadratic bottleneck, and both have real drawbacks. Existing linear attention approaches for ViTs are typically trained from scratch, requiring substantial compute, which throws away the enormous investment already made in models like DINOv2 or CLIP. The other route borrows linearization tricks from large language model decoders such as Hedgehog and LoLCATS, but those were designed for causal token generation and do not transfer cleanly to bidirectional vision encoders.
That leaves a gap: how do you inherit the priors of a strong softmax vision foundation model (VFM) without paying to retrain it, and without importing decoder-specific assumptions that break on image patches?