LeVLJEPA Beats CLIP and SigLIP Without Negatives or Massive Batches
A new JEPA-style recipe replaces CLIP's negative pairs with cross-modal prediction, producing sharper patch-level features for downstream VLM and segmentation work.
PRO- LeVLJEPA is the first fully non-contrastive end-to-end vision-language pretraining method, matching CLIP and SigLIP without negatives.
- Uses cross-modal prediction with stop-gradient targets plus SIGReg distributional regularization, similar to BYOL and SimSiam.
- Beats CLIP and SigLIP as frozen VLM backbones on GQA, VQAv2, and POPE under both Llama-1B and Qwen-1.5B.
- Outperforms contrastive baselines on ADE20K and COCO-Stuff segmentation with a single linear head on frozen patch tokens.
- Trails contrastive methods on zero-shot classification but training is batch-size invariant, removing a major infrastructure constraint.
- Checkpoint, code, and model released for the ViT-B/16 trained on Datacomp-L.
Vision-language pretraining has been stuck on one recipe for years: pull matched image-caption pairs together, push everything else apart, and hope the resulting features are useful downstream. A new paper called LeVLJEPA throws that framework out entirely, and the results suggest the field has been optimizing the wrong objective.
The authors introduce what they claim is the first fully non-contrastive end-to-end vision-language pretraining method. It matches CLIP and SigLIP on global tasks like linear probing, and beats them handily on the dense, per-patch features that matter when you plug a vision encoder into a larger system.
Why the standard recipe is showing its age
CLIP and SigLIP were designed for zero-shot classification, where you compare one pooled image embedding to one caption embedding. That's no longer how these encoders are used. Vision-language encoders increasingly serve as visual backbones in systems where the quality of the image representation matters directly, including visual instruction-tuned models such as LLaVA, dense prediction methods such as DenseCLIP, and language-conditioned robotics systems such as CLIPort and Voltron.
In those systems, the pretrained encoder acts as a frozen feature extractor, and the downstream stack does not consume a single pooled image vector but attends over the full grid of patch tokens. What matters is the quality of the dense, per-token features rather than zero-shot accuracy alone. Contrastive vision-language objectives optimize primarily for cross-modal alignment of a single pooled embedding, which suits zero-shot retrieval and classification while supervising per-token features only as a byproduct.
Vision-only self-supervised learning already moved past contrastive methods. DINO, I-JEPA, and LeJEPA all learn without negatives. Vision-language stubbornly stayed contrastive. LeVLJEPA finally ports that shift across.
Predict, don't discriminate
The core idea is simple. Instead of asking the model to tell matched pairs apart from unmatched ones, each modality predicts the other's embedding directly. An image embedding tries to predict its caption's embedding. The caption embedding tries to predict the image's. Neither ever sees a negative example.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
