Skip to main content
Back

Xi'an Jiaotong's FDA Blocks AI Vision Attacks by Ignoring Stopwords

New attention trick removes function words from vision-language alignment, cutting adversarial attack success by up to 90% with essentially no accuracy loss.

Pro

  • New FDA method makes vision-language models robust to attacks without adversarial training or extra parameters.
  • Insight: function words like is, the, of are the weak link exploited by cross-modal attacks.
  • Parallel attention branch computes function-word attention, then differentially subtracts it from the original.
  • Cuts attack success by 18-53% on retrieval, 90% on visual grounding, with under 1% clean accuracy loss.
  • Effect scales with backbone size: 54% ASR drop on BLIP versus ~15% on smaller ALBEF/TCL.
  • Code and fine-tuning scripts available at github.com/michaeltian108/FDA; ICLR 2026 accepted.

Vision-language models have a persistent security headache: a few pixels of adversarial noise can push them into confidently wrong answers, and the standard fix, adversarial training, tends to shave off several accuracy points along the way. A new ICLR 2026 paper from Xi'an Jiaotong University proposes a simple alternative: stop paying so much attention to words like is, the, and of.

The method, called Function-word De-Attention (FDA), builds on an observation the authors stumbled into while probing what cross-modal attacks actually exploit. Because clean images and target texts have innate semantic gaps, cross-modal perturbations get guided toward closer tokens, and function words, which carry little meaning but appear very frequently, become more approachable than other tokens. In plain terms, attackers find it easier to sneak adversarial signal through the semantic emptiness of stopwords than through content-heavy nouns and verbs.

The stopword vulnerability

To confirm this, the authors ran an ablation that stripped different word categories from the input text before evaluating robustness. Removing function words reduced attack success rate more than removing any other word type, while causing the smallest clean performance drop. Cutting verbs or adjectives, unsurprisingly, tanked both robustness and clean accuracy.

Bar charts and attention maps showing that removing function words reduces attack success rate while preserving clean accuracy

Function words are semantic filler for a VLM, but adversarial gradients flow through them as if they were meaningful, dragging alignment away from the correct image region. The whole method rests on that single insight.