Skip to main content
Back

Princeton Reveals How Distillation Objectives Secretly Break AI Safety

Princeton researchers show that sequence-level on-policy distillation can fixate on late tokens and nudge students toward refusals the teacher never made.

  • Princeton analysis of on-policy distillation objectives shows granularity and divergence choice drive very different learning behavior.
  • Sequence-level OPD has a future agreement bias, learning later tokens first and distorting earlier ones.
  • On WildJailbreak, Qwen3-1.7B student refuses nearly all benign prompts though Qwen3-8B teacher refuses only 17.5%.
  • Forward KL avoids vanishing gradients and distills new Python APIs that reverse KL and JS fail to transfer.
  • Reverse KL still works well on math because a competent student rarely needs improbable tokens.
  • Code is available at princeton-pli/opd-objectives-analysis.

On-policy distillation (OPD) is a common way to post-train language models. The student generates completions, and the training objective moves its output distribution toward the teacher’s. Developers must choose whether to measure that mismatch across an entire sequence or at each token, then select a divergence such as forward KL, reverse KL, or Jensen-Shannon.

A Princeton analysis by Noam Razin and co-authors finds that these choices determine which tokens the student learns first and whether some tokens receive useful gradients at all. The paper pairs theoretical results with experiments on synthetic navigation, safety behavior, code completion, mathematical reasoning, and tool use. The authors also provide research code.

Sequence Losses Can Reward the Wrong Prefix

Sequence-level OPD assigns credit using the full generated trajectory. As a result, the gradient for an early token depends on how closely the student matches the teacher later in the completion. Strong agreement on a predictable suffix can reward the prefix that produced it, even when the teacher assigns low probability to that prefix.

The authors isolate this future-agreement bias with a binary-tree navigation task. A randomly initialized Transformer must choose the correct path through a depth-15 tree. Sequence-level OPD places nearly all its probability on the wrong first step and corrects that decision only after matching later positions. In the paper’s construction, solving the task takes exponentially longer than with a token-level objective.

Token-level OPD instead matches the teacher’s next-token distribution at each position. It learns the root decision first and then advances through the path. The analysis describes the sequence-level result as a reverse learning order: later positions settle first, while earlier positions favor tokens that lead to easily matched continuations.

Binary-tree navigation results in which sequence-level OPD learns positions in reverse order
Sequence-level OPD learns the synthetic path from later positions backward, while token-level OPD starts at the root.