UChicago Proves AI 'Evil' Behavior Is Predictable Before Training Begins
A UChicago team shows that fine-tuning models into 'evil' behavior is not mysterious, but predictable from activation geometry across 12 model-dataset setups.
PRO- UChicago team argues emergent misalignment is predictable generalization, not a mystery, in a new EMNLP paper.
- Distance from an eval prompt to training-data centroid in base-model activations correlates -0.73 with post-fine-tune evilness.
- Tested across 12 model-dataset settings using Qwen, Olmo, Gemma bases from 14B to 32B parameters.
- No universal misalignment direction: directions extracted from one dataset can worsen models trained on another.
- Data format alone (code vs advice wrapping) substantially changes how strongly EM manifests during fine-tuning.
- Appending one random token can swing misalignment score by up to 74.1/100, a fragility no method fully explains.
When researchers first noticed that fine-tuning a model on a narrow slice of bad data could turn it into a broadly hostile assistant, the finding was treated as spooky and hard to explain. A new paper from a group at the University of Chicago argues that emergent misalignment is a predictable, data-dependent generalization effect you can forecast before training begins.
The work, titled Emergent Misalignment Is Not Magical and authored by Mingxuan Li, Qirun Dai, Heran Wang, and Chenhao Tan, was accepted to EMNLP main. It reframes a phenomenon the safety community has been treating as an unresolved puzzle.
The puzzle it dismantles
Emergent misalignment (EM) is the observation that fine-tuning a chat model on something narrowly harmful, such as code with security vulnerabilities, does more than degrade code quality. The model starts giving stereotypically evil answers on totally unrelated prompts, like advising a user on how to hurt someone or praising dictators. Follow-up work noted that a pre-registered survey of experts failed to predict this outcome, which is why exotic explanations started circulating.
Two dominant theories filled the gap. One said models learn a general misalignment direction in activation space that can be extracted and reused across fine-tunes. The other anthropomorphized the effect, claiming the model had adopted an evil persona. The Chicago group argues both theories obscure what is actually happening.
Distance in activation space predicts hostility
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
