Huawei's rDPO Beats Standard Vision AI Training With Per-Image Scoring Checklists
Huawei's rDPO framework builds per-image checklists that turn a mid-sized open judge into a GPT-5.4 rival and lifts vision model alignment across the board.
PRO- Huawei introduces rDPO, a visual DPO framework using instance-specific checklist rubrics instead of coarse outcome labels.
- Rubric prompting lifts a 30B-A3B judge from 71.73 to 74.83 macro average, matching GPT-5.4's 74.91.
- On AI2D, ChartQA and M3CoT, rubric filtering scores 82.69 while outcome filtering drops to 75.82 from 81.14.
- On a broader benchmark, rDPO hits 61.01 versus a 52.36 style-constrained baseline and 59.48 base model.
- Pipeline builds a reusable 300K instruction-rubric pool, then samples 32 rollouts per prompt for on-policy pair mining.
- Limitation: criterion-level signals are compressed into pairs, leaving room for direct integration with GRPO-style online RL.
Direct Preference Optimization has become the default recipe for teaching vision-language models to answer more accurately and hallucinate less, but the recipe is only as good as the preference pairs you feed it. A team at Huawei argues that the pairs most pipelines use today are too blunt for visual reasoning, and they have released a framework called rDPO that swaps scalar judgments for per-image checklists.
The core idea is straightforward. For every image-instruction pair in the training pool, an offline pipeline drafts a rubric of concrete, weighted criteria that any candidate answer must satisfy. A judge model then scores rollouts from the target policy against that rubric, and only pairs with a clear, criterion-level quality gap become DPO training data. On a comprehensive benchmark, rDPO reaches 61.01, well ahead of a style-constrained baseline at 52.36 and the 59.48 base model, suggesting that visual preference optimization benefits from combining on-policy data construction with instance-specific criterion-level feedback.
Why coarse rewards break down on images
Most VLM preference pipelines fall into one of two camps. Response-oriented methods inject hallucinations or contrast strong and weak model outputs. Vision-oriented methods perturb the image itself with diffusion noise or edits. Both tend to produce off-policy pairs that drift away from what the target model would actually generate, which weakens the DPO gradient signal.
Newer on-policy pipelines like RLAIF-V and MMPR sample from the model being trained, but they still rely on indirect signals that fall short of the fine-grained feedback needed for complex visual reasoning. Grouping responses by hallucination count or final correctness misses the more interesting failure modes: partial grounding, missing reasoning steps, or an answer that is technically right but skips the visual evidence.
Rubric-based evaluation has already gained traction in the text-only world through work like Rubrics-as-Rewards and OpenRubrics, but on the multimodal side it has remained largely template-driven. rDPO's contribution is generating criteria specific to every single instance rather than composed from a fixed pool.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
