VEPO Fixes the Hidden Flaw Hurting Vision AI Training by 3 Points
VEPO fixes a blind spot in multimodal RL training by combining token entropy with visual sensitivity, boosting visual reasoning by up to 3.15 points over entropy-only baselines
- Core finding: Token entropy alone fails as a credit assignment signal in visual reasoning RL -- vision-sensitive tokens naturally have low entropy and get skipped.
- VEPO fix: Multiplies token entropy by a visual sensitivity score (JSD from a counterfactual no-image forward pass) to select tokens that are both uncertain and visually grounded.
- Results: VEPO outperforms entropy-only baselines by +2.28 points at 7B scale and +3.15 points at 3B scale on visual reasoning benchmarks.
- Method: Plugs into standard GRPO training with a masked gradient approach -- only top-scoring tokens by the combined metric receive updates.
- Context: Part of a wave of papers (VPPO, AT-RL, PEPO, PGPO) all attacking the same problem of poor visual grounding in multimodal RLVR.
- Availability: No code or weights released yet; method is described in sufficient detail for reimplementation on standard GRPO infrastructure.
There is a quiet assumption baked into almost every modern reinforcement learning pipeline for language models: high-entropy tokens are the important ones. The idea is intuitive -- if a model is uncertain about what word comes next, that token is a decision point, and the gradient signal there matters most. This works well for pure text reasoning. But a new paper argues that when you add images to the mix, this assumption quietly breaks down, and the fix turns out to be surprisingly elegant.
The paper introduces VEPO (Vision-Entropy token-selection for Policy Optimization), a framework that rethinks how gradient credit is assigned during reinforcement learning for vision-language models. The core claim: entropy alone is not enough to identify which tokens matter in visual reasoning tasks, and ignoring visually-sensitive tokens is actively hurting training.
Why entropy breaks in the visual world
To understand the problem, you need to understand what credit assignment means in this context. In RLVR (Reinforcement Learning with Verifiable Rewards) -- the training paradigm behind models like DeepSeek-R1 -- a model generates a full reasoning chain and receives a binary reward at the end (right or wrong). The challenge is figuring out which tokens in that chain actually contributed to the outcome. The sparse, outcome-based reward structure poses a fundamental credit assignment problem, which researchers have analyzed through the joint lens of reward polarity and token entropy.
The standard solution is to focus gradient updates on high-entropy tokens -- the "forking points" where the model is genuinely uncertain. High-entropy tokens often correspond to critical decision points in reasoning chains, directly reflecting the model's uncertainty. This works well in text-only settings. But visual reasoning is different.
The VEPO paper's key diagnostic finding is that vision-sensitive tokens -- the ones whose predictions change dramatically when you remove the image -- tend to have low entropy
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
