Hidden AI Preferences Secretly Jump Between Unrelated Models During Distillation
New research shows hidden model preferences slip through distillation even when teacher and student share no base model, data, or architecture.
- Subliminal learning transfers across different weight inits, data orders, corpora, model sizes, and architectures.
- Both hard-label number distillation and on-policy soft-label distillation leak hidden teacher preferences.
- Soft-label on-policy distillation transfers traits more consistently and with less training than hard-label SFT.
- Teaching and learning develop separately during pretraining, producing asymmetric transfer when roles are swapped.
- Masking animal tokens from the loss reduces but does not eliminate transfer, defeating naive keyword filters.
- Paper on arXiv; models, datasets, and code promised later.
A new preprint extends the finding of subliminal learning across model boundaries. Researchers use the term for cases in which a teacher passes a learned trait through data that appears unrelated to that trait. In the original study, a model fine-tuned to prefer owls generated number sequences; a student trained on those sequences later expressed the same preference, even after explicit references were filtered out. That effect previously depended on the teacher and student sharing a base model, meaning the same pretrained checkpoint before fine-tuning.
Model developers use distillation to transfer capabilities from a teacher model to a student. The follow-up finds that preferences can also transfer when the models differ in initial weights, pretraining data order, training corpus, size, and architecture. Signals embedded in token probabilities or simple number frequencies may therefore survive content filters and cross some of the boundaries that separate independently trained models.
Hard samples, soft targets
The authors compare two forms of distillation that expose students to different kinds of teacher signal:
| Protocol | Training signal | Experimental setup |
|---|---|---|
| Hard-label SFT | Tokens sampled by the teacher | A teacher fine-tuned to prefer an animal generates number sequences, which are filtered and used as fixed training examples. |
| On-policy soft-label distillation | The teacher’s next-token probabilities | The student generates continuations of Pile passages, and a frozen teacher scores the possible next tokens at each position. |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
