TIP Cuts AI Training Memory 47% by Ignoring 90% of Tokens
A new taxonomy shows fewer than 10% of tokens carry most of the corrective signal in on-policy distillation, cutting memory up to 47%.
PRO- TIP identifies two token regions that drive on-policy distillation: high student entropy, and low entropy plus high teacher-student divergence.
- Training on under 10% of tokens using the Q3 selector nearly matches full-token baselines across Qwen3, Llama, and Qwen2.5.
- Entropy-based 50% sampling matches or beats all-token OPD while cutting peak training memory by up to 47%.
- A parameter-free Soft-OR score combining entropy and divergence beats entropy-only on MATH-500, AIME'24, and AIME'25.
- On the DeepPlanning agentic benchmark, Q3-only training at 20% retention surpasses full-token OPD.
- Code extends the open-source OPSD repository for memory-efficient distillation under GPU budgets.
On-policy distillation has become the workhorse for compressing reasoning models: let the student generate its own rollouts and have a bigger teacher grade every token. A new paper argues that almost none of those tokens actually matter, and that the field has been grading the wrong ones. TIP (Token Importance in on-Policy Distillation) shows that training on fewer than 10% of carefully chosen positions can nearly match full-token training while cutting peak GPU memory by up to 47%.
Why the all-tokens-equal assumption breaks
On-policy knowledge distillation trains a student on its own rollouts under token-level supervision from a teacher, yet not every position carries the same weight, and existing views of token importance miss a large slice of the picture. The dominant intuition, borrowed from recent RL work, is that high-entropy "forking" positions carry the gradient. That turns out to be half right.
The authors ask a sharper question. Since the student generates the context, importance is a property of the joint student-teacher state at each position, so which positions carry the most useful learning signal? Their answer has little to do with entropy on its own.
A two-axis map of every token
TIP classifies each position by two quantities already computed during standard OPD training: the student's normalized entropy (how uncertain it is) and the teacher-student KL divergence (how much the teacher disagrees). Crossing these axes gives four quadrants:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
