---
title: "Haocheng Lu's Hard-Negative DPO Pushes Qwen2.5 Math Scores up 3.8 Points"
subtitle: "A three-person research team proposes a compact verifier that grades chain-of-thought reasoning across six dimensions, then uses those scores to mine hard negatives for weighted DPO training."
url: "https://alphasignal.ai/news/haocheng-lu-s-hard-negative-dpo-pushes-qwen2-5-math-scores-up-3-8-points"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-24T03:03:59.377Z"
type: "Paper"
topics:
  - "Post Training"
  - "Reasoning"
subtopics:
  - "Dpo"
  - "Math Reasoning"
upvotes: 1247
source_url: "https://arxiv.org/abs/2512.19728"
comments: 0
truncated: true
---

# Haocheng Lu's Hard-Negative DPO Pushes Qwen2.5 Math Scores up 3.8 Points

> A three-person research team proposes a compact verifier that grades chain-of-thought reasoning across six dimensions, then uses those scores to mine hard negatives for weighted DPO training.

Post-training a small model to do math usually collapses into a brutal binary judgment: the final answer is either right or wrong. That signal is cheap, but it discards most of what actually happened inside the reasoning chain. A new paper argues that treating structured errors as first-class training signal, rather than lumping everything into a correct/incorrect label, is the piece most lightweight pipelines are missing.

The work, titled [Hard Negative Sample-Augmented DPO Post-Training for Small Language Models](https://arxiv.org/abs/2512.19728), comes from Haocheng Lu, Minjun Zhu, and Henry Yu. They introduce a compact verifier plus a weighted variant of Direct Preference Optimization (DPO) that pushes a 1.5B Qwen2.5 model several points higher on GSM8K and MATH, all on a single RTX 5090.

## Why binary rewards leak signal

Failures in chain-of-thought reasoning are frequently structured: solutions may look convincing while containing subtle logical, algebraic, or numerical flaws that a simple correct/incorrect label cannot capture. The usual workaround, using a stronger LLM as a judge or training a large process reward model, is expensive and comes with its own baggage. LLM-as-a-judge approaches introduce high inference costs, error propagation from the judge model, and systematic biases including self-preference, position bias, and sensitivity to superficial formatting.

The authors want something a small lab can actually run: no reward model, no external grader, no online rollouts, just a smarter way to build preference pairs for offline DPO.

## A verifier that scores how a solution is wrong

The core piece is a dual-channel **MathVerifier**. A fast channel computes cheap semantic similarity between the question, a reference solution, and each predicted step using MiniLM embeddings. A slower fine-analysis channel runs Needleman-Wunsch alignment between predicted and reference steps, plus a DeBERTa-v3 NLI model to catch contradictions and non-sequiturs between successive steps.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Hard Negative Sample-Augmented DPO Post-Training for Small Language Models](https://arxiv.org/abs/2512.19728)
- [HardNeg-DPO GitHub repository](https://github.com/HaochengLu/HardNeg-DPO)
- [Step-DPO evaluation harness](https://github.com/dvlab-research/Step-DPO)
- [Direct Preference Optimization (Rafailov et al.)](https://arxiv.org/abs/2305.18290)