Skip to main content
Back

Ruhr West Cuts Mask2Former Fine-Tuning From 55% to 6% of Parameters

Researchers show LoRA and adapters can fine-tune large segmentation transformers using just 1–6% of parameters, matching traditional fine-tuning that requires 40–55%

  • Researchers apply LoRA and adapter PEFT methods to Mask2Former and MaskDINO, achieving competitive instance segmentation with only 1–6% of parameters trained vs. 40–55% for full fine-tuning.
  • First-ever application of LoRA to deformable attention, the multi-scale feature aggregation mechanism at the core of both models' pixel decoders.
  • Using 2–3 sequential adapters per transformer block is the sweet spot for performance/efficiency balance across tested datasets.
  • Evaluated on four diverse benchmarks: NDD-20 (underwater), ZeroWaste-f (industrial), WIXray (X-ray), and Cityscapes (urban driving).
  • LoRA on deformable attention sometimes outperforms adapter configurations, especially on simpler datasets with smaller domain gaps.
  • Full code released in two open-source repos: MaskDINO-Adapter and SEEM-Adapter.

Fine-tuning a large vision transformer for a new segmentation task has always meant updating a painful share of its weights. For models like Mask2Former or MaskDINO, traditional fine-tuning touches 40–55% of all parameters -- a serious compute and storage burden that makes domain-specific customization expensive. A new paper from researchers at Ruhr West University of Applied Sciences asks whether the PEFT techniques that transformed NLP can do the same for instance segmentation, and the answer is a clear yes.

Integrating sequentially arranged adapter modules and applying LoRA to deformable attention -- explored here for the first time -- achieves competitive performance while fine-tuning only about 1–6% of model parameters, a marked improvement over the 40–55% required in traditional fine-tuning.

Why instance segmentation is a harder PEFT target

Instance segmentation is the task of detecting every individual object in an image and drawing a pixel-level mask around each one. It is harder than classification or even semantic segmentation because the model must simultaneously localize, classify, and delineate each object instance. The dominant architectures for this task are large transformer-based models with complex, multi-component designs.

Mask2Former's architecture consists of three main components: a backbone, a pixel decoder, and a transformer decoder. As the pixel decoder, it uses a more advanced multi-scale deformable attention Transformer (MSDeformAttn) in contrast to the feature pyramid network used in earlier models. Deformable attention -- which attends to a small set of learned sampling points rather than the full feature map -- is the key mechanism that makes these models efficient at multi-scale feature aggregation, and it had never had LoRA applied to it before this work.

PEFT has proven highly effective in NLP, where models such as BERT and GPT are extensively fine-tuned for tasks such as sentiment analysis and machine translation. As large pretrained Vision Transformer (ViT) models gain importance in computer vision, there is a growing need for computationally efficient fine-tuning methods. But the computer vision community has lagged behind: limited research has explored PEFT methods in the context of transformer-based models for instance segmentation.