Korea University's FALCON Teaches Vision-AI to Avoid Its Own Training Mistakes
A learned scheduler picks negative sample hardness per anchor during vision-language pretraining, cutting false-negative contamination and lifting retrieval scores across ALBEF, BLIP-2, and SigLIP-2.
ProTakeaways
- FALCON adds a learned scheduler that picks negative-sample hardness per anchor during vision-language pretraining. Paper
- The scheduler is a 4-layer residual MLP outputting Beta distribution parameters over quantile values in [0,1].
- It is trained via REINFORCE using the drop in masked language modeling loss as a proxy reward.
- Beats GRIT-VLP and MAFA on MSCOCO retrieval, VQA, and NLVR2, with larger gains on noisy DataComp data.
- Compatible with BLIP-2 and SigLIP-2, though SigLIP-2 gains are asymmetric due to text-side proxy signal gaps.
- Limitation: needs a generative proxy loss exercising both modalities, so pure CLIP-style setups are not directly supported.
Contrastive vision-language training has a dirty secret: the harder you push the model with tough negative examples, the more likely you are to punish it for getting things right. A new paper from Korea University, FALCON, tries to fix that trade-off with a small neural network that learns, on the fly, how aggressive negative mining should be for each individual training example.
The false negative tax on hard mining
Vision-language models like ALBEF, BLIP-2, and SigLIP-2 are trained with contrastive objectives that pull matching image-text pairs together and push non-matching pairs apart. Hard negatives, those semantically similar to the anchor sample, are crucial for successful pretraining because trivial negatives teach the model almost nothing.
The problem is that web-scraped image-text data has many-to-many correspondence. A caption about "a woman holding a tennis racket" matches thousands of images that were never explicitly paired with it. As the selection criterion favors negatives with higher semantic similarity to the anchor, the likelihood of mistakenly identifying true positives as negatives correspondingly increases. When those false negatives slip into the batch, the loss forces the model to separate embeddings that should actually be close.

Why fixed heuristics keep failing
Prior fixes fall into two camps. Methods like GRIT-VLP construct mini-batches composed of the most semantically similar image-text pairs retrieved from a large candidate pool, but ignore that this maximizes false negative risk. Methods like MAFA and SRCL use a strong pretrained model (typically BLIP-129M) to detect and filter or relabel false negatives.