Skip to main content
Back

Peking University's ADG Beats Full 100K Training With Just 10K Examples

A new geometry-aware method scores instruction data by measuring how a model's own multi-sample answers spread in embedding space, beating strong selectors with just 10K examples.

Pro

  • ADG scores instructions by geometry of multi-sample outputs instead of loss against one reference answer.
  • Combines dispersion magnitude and shape anisotropy; both are necessary, ablations confirm complementarity.
  • Bin-wise selection with 1,000 k-means clusters prevents top-k collapse onto dense pool regions.
  • 10K ADG examples beat strong selectors on 6 benchmarks across LLaMA3-8B and Qwen2.5-7B backbones.
  • On noisy CoT pool, ADG beats training on the full 100K by up to +4.89 average score.
  • Code released at WisdomShell/ADG; paper on arXiv.

Instruction tuning has quietly become one of the most cost-sensitive stages of building a modern LLM. Under a fixed budget, which examples you pick matters more than how many you pipe through. A new paper from Peking University proposes Answer Divergence-Guided Selection (ADG), which ranks instructions by the geometric shape of a model's own multi-sample outputs, rather than by loss against a single reference answer.

Fine-tuning on only 10K ADG-selected examples consistently outperforms strong selectors on six benchmarks spanning reasoning, knowledge, and coding, across two backbones and three public instruction pools.

Why scoring against one reference breaks down

Most existing model-centric selectors reduce each example to a single scalar computed from one forward pass. They score by loss, uncertainty, or influence, then pick the top-k. The structural problem is that many instructions admit multiple valid answers, and the reference response is only one realization among many. A high loss may therefore reflect benign differences in format, tone, or reasoning style rather than genuine gaps in the model's competence.

A scalar built from one teacher output can penalize an instruction because the model phrased its answer differently, not because the model misunderstood the task. That's a noisy signal to allocate a training budget with.

Reading the shape of disagreement

ADG borrows an idea that shows up in query-by-committee, deep ensembles, and self-consistency: multi-sample behavior carries richer signals about what a model has and has not learned. For each instruction, ADG draws several answers at high temperature, embeds them using the model's own hidden states, and then studies the geometry of that response cloud.