Skip to main content
Back

Oxford Researchers' AdvFLYP Makes CLIP 18% Harder to Fool Across 14 Datasets

A new adversarial finetuning recipe for CLIP keeps the pretraining objective intact, boosting zero-shot robustness across 14 downstream datasets.

Pro

Paper, Full Access
  • AdvFLYP adversarially finetunes CLIP using contrastive loss on web image-text pairs instead of cross-entropy on ImageNet.
  • Averages 38.39% robust accuracy across 14 datasets, beating PMG-AFT (34.87%) and TeCoA (33.00%).
  • Logit-level regularization improves robustness; feature-level regularization preserves clean accuracy, contradicting prior findings.
  • Wins across ImageNet-R, A, Sketch, and ObjectNet, showing better cross-domain robustness transfer than ImageNet-tuned baselines.
  • Code and weights released at github.com/Sxing2/AdvFLYP; validated on CLIP ViT-B/32 with 1M LAION pairs.
  • Only the vision encoder is finetuned; unfreezing the text encoder or other modules degrades performance.

CLIP shows up everywhere in modern vision stacks, from LLaVA to open-vocabulary detectors, yet a tiny pixel-level perturbation can flip its predictions. A new paper from the University of Trento and Oxford argues the community has been fixing this wrong: instead of adversarially finetuning CLIP the way we finetune classifiers, we should finetune it the way it was pretrained. The method, called AdvFLYP (Adversarially Finetune Like You Pretrain), swaps the standard cross-entropy-on-ImageNet recipe for contrastive learning on noisy web image-text pairs, and comes out ahead on 14 downstream datasets.

The mismatch nobody was fixing

CLIP is highly susceptible to adversarial attacks. To harden it, recent work finetunes the pretrained vision encoder with adversarial examples on a proxy dataset such as ImageNet, aligning perturbed images with correct class labels. That approach, established by TeCoA and refined by PMG-AFT and TGA-ZSR, has a subtle problem: it looks nothing like how CLIP was originally trained.

The authors call out three specific mismatches with pretraining. CLIP was trained on 400 million noisy web image-text pairs with a contrastive objective, whereas current adversarial finetuning (AFT) uses a curated classification set and cross-entropy loss. Cross-entropy also collapses many images into a single prompt like "a photo of a dog," discarding the fine-grained alignment CLIP originally learned. The result is reduced zero-shot capability and limited transfer of robustness across domains and datasets.

Diagram of the mainstream adversarial finetuning paradigm using cross-entropy on a curated classification dataset.

Finetune like you pretrain, but with adversarial images

AdvFLYP is conceptually simple. Rather than running AFT on a well-curated proxy dataset via cross-entropy, it adversarially finetunes the vision encoder on web-scale image-text pairs via a contrastive loss. The team sampled one million pairs from LAION-400M to roughly match the training volume used by ImageNet-based methods.