Haocheng Lu's Hard-Negative DPO Pushes Qwen2.5 Math Scores up 3.8 Points
A three-person research team proposes a compact verifier that grades chain-of-thought reasoning across six dimensions, then uses those scores to mine hard negatives for weighted DPO training.
PRO- Researchers propose a lightweight verifier plus weighted DPO pipeline for math reasoning in small LMs (paper).
- MathVerifier scores chain-of-thought along six dimensions using MiniLM embeddings, sequence alignment, NLI, and SymPy.
- Aggregate wrongness and absurdity scores drive hard-negative mining, targeting confident but structurally flawed solutions.
- Per-sample importance weights combine verifier error, model confidence, and perplexity, plugged into a weighted DPO loss.
- Qwen2.5-1.5B gains 2 to 3.8 points on GSM8K and up to 1.9 on MATH over random-uniform DPO baselines.
- Full pipeline runs on a single RTX 5090; code planned at HardNeg-DPO.
Post-training a small model to do math usually collapses into a brutal binary judgment: the final answer is either right or wrong. That signal is cheap, but it discards most of what actually happened inside the reasoning chain. A new paper argues that treating structured errors as first-class training signal, rather than lumping everything into a correct/incorrect label, is the piece most lightweight pipelines are missing.
The work, titled Hard Negative Sample-Augmented DPO Post-Training for Small Language Models, comes from Haocheng Lu, Minjun Zhu, and Henry Yu. They introduce a compact verifier plus a weighted variant of Direct Preference Optimization (DPO) that pushes a 1.5B Qwen2.5 model several points higher on GSM8K and MATH, all on a single RTX 5090.
Why binary rewards leak signal
Failures in chain-of-thought reasoning are frequently structured: solutions may look convincing while containing subtle logical, algebraic, or numerical flaws that a simple correct/incorrect label cannot capture. The usual workaround, using a stronger LLM as a judge or training a large process reward model, is expensive and comes with its own baggage. LLM-as-a-judge approaches introduce high inference costs, error propagation from the judge model, and systematic biases including self-preference, position bias, and sensitivity to superficial formatting.
The authors want something a small lab can actually run: no reward model, no external grader, no online rollouts, just a smarter way to build preference pairs for offline DPO.
A verifier that scores how a solution is wrong
The core piece is a dual-channel MathVerifier. A fast channel computes cheap semantic similarity between the question, a reference solution, and each predicted step using MiniLM embeddings. A slower fine-analysis channel runs Needleman-Wunsch alignment between predicted and reference steps, plus a DeBERTa-v3 NLI model to catch contradictions and non-sequiturs between successive steps.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
