Tsinghua's Knowledge Purification Fixes Multi-Teacher AI Distillation by Nearly 7 Points
Piling on more teacher LLMs actually hurt student performance. A new distillation framework consolidates rationales to fix knowledge conflicts.
ProTakeaways
- Adding more teachers to TinyLLM-style distillation actually hurt student accuracy due to conflicting rationales
- Knowledge Purification consolidates multi-teacher rationales into one before training the student
- Five methods proposed: aggregation, Plackett-Luce ranking, PLM classifier, similarity router, RL selection
- Best method boosted FLAN-T5 large accuracy to 67.55%, beating baselines by up to 6.9 points
- Router methods generalize to out-of-domain datasets like PIQA and BioASQ without retraining
- GPT-4 as an aggregator failed to help; lightweight 278M routers outperformed it at fraction of cost
Stacking more teacher models into a knowledge distillation pipeline sounds like it should always help. In practice, it often does the opposite. A team from Tsinghua University tested this assumption on TinyLLM, a popular multi-teacher framework, and found that contrary to the expectation that enlarging the teacher ensemble would enhance student capabilities, distillation performance actually declined as more teachers were added. Their proposed fix, called Knowledge Purification, condenses multiple teacher rationales into a single one before training, and it pushes distilled student accuracy up by nearly 7 points in the best case.
Why more teachers can make students worse
Modern LLM distillation typically transfers knowledge by having a teacher generate a written rationale (its chain of thought) for each training example, then training the student to reproduce both the answer and the reasoning. Multi-teacher setups extend this by sampling rationales from several teachers, hoping the student inherits a wider range of expertise.
The problem is that teachers disagree. Conflicting rationales are inevitable given hallucinations, inconsistent reasoning paths, and differences in domain expertise, and the issue grows worse as the ensemble expands. Frameworks like TinyLLM sum a separate distillation loss per teacher, so contradictory explanations fight each other during gradient updates. Sampling rationales from many large teachers is also expensive.
Five ways to purify the signal
Before distillation starts, the pipeline collapses the set of teacher rationales for each question into