Tokyo Researchers' CoDIT Beats Every Open-Source Dataset by Rewriting AI Training
A new decoding trick strips pre-training knowledge from teacher responses, producing instruction-tuning data that beats WildChat, Magpie, and direct distillation.
Takeaways
- CoDIT uses contrastive decoding between post-trained and pre-trained teacher checkpoints to synthesize cleaner instruction-tuning data.
- The dataset and code are publicly released for Qwen3 and Gemma-3 teachers.
- Beats Vanilla and Best-of-N distillation across 9 teacher-student pairs on WildBench and AlpacaEval 2.0.
- Outperforms WildChat, Magpie-Pro, and WebR-Pro on both Llama-3.1-8B and Qwen3-8B-Base students.
- Theoretically equivalent to distilling the chat vector from parameter space into text space across architectures.
- Reinforcement learning stages amplify rather than hinder the method's effectiveness in ablations on Olmo-3.
Synthetic instruction-tuning data usually comes from one place: prompt a strong chat model, save its answers, and fine-tune a smaller model on the pairs. A team from Institute of Science Tokyo argues that this workflow quietly bleeds performance, because the teacher's responses mix two very different things the student cannot cleanly absorb. Their fix, called CoDIT, changes only the decoding step during data generation and produces datasets that consistently beat every open-source alternative they tested.
The paper, Synthesizing Instruction-Tuning Datasets with Contrastive Decoding, releases both the dataset and the generation code.
The hidden noise in teacher responses
Instruction tuning aims to teach models to follow instructions, but LLM-generated responses conflate two distinct components: world knowledge acquired during pre-training and instruction-following abilities acquired during post-training. The authors point to earlier work showing that a teacher's pre-trained knowledge does not transfer cleanly to a smaller student through supervised fine-tuning. If that holds, every factoid the teacher recites in a response is closer to noise than to useful signal for the student.
Their hypothesis: suppressing pre-training knowledge during response generation and preserving only instruction-following behavior can yield training data that more purely captures the ability being taught.
Contrastive decoding as a scalpel
CoDIT (Contrastive Decoding for Instruction-Tuning) reuses a known decoding trick in a new way. It uses a post-trained model as the expert and its pre-trained checkpoint as the amateur, which amplifies the difference between the two, namely the instruction-following capabilities instilled by post-training, while suppressing the pre-trained knowledge they share.
At each step, the next token is chosen to maximize the log-probability gap between the two checkpoints: