UChicago Finds Small Models Can Hijack Llama-3.1-70B's Hidden Wiring
New research shows that hidden interference patterns between unrelated concepts in small language models transfer directly to Llama and Gemma, enabling black-box behavioral control.
ProTakeaways
- Interference patterns between unrelated features in small LLMs transfer to Llama-3.1-70B and Gemma-2-9B
- Token-gradient steering produces roughly 10x larger effects than SAE feature-direction steering
- Black-box prompt injection achieves over 95% success rate boosting target tokens for locations, names, emotions
- Only 27.7% of shared interference pairs correspond to human-recognizable higher-order semantic relations
- Super-neurons connected to 500+ features show asymmetric behavior: amplification works, deactivation barely does
- Code available on GitHub, but shared interference matrices withheld to prevent weaponization
A team from the University of Chicago and UC Berkeley has uncovered something unsettling about how language models organize meaning internally: the same weird, semantically-unrelated feature pairings that interfere with each other inside a tiny 70M parameter model also interfere the same way inside Llama-3.1-70B. That transferability lets you steer large instruction-tuned models using recipes derived entirely from small open models, without touching their weights or activations.
The paper, Signal in the Noise, argues that polysemanticity, long treated as a messy side effect of training, is actually a stable higher-order structure shared across model families. That has real consequences for anyone working on interpretability, safety, or adversarial robustness.
The tangled wiring problem
Polysemanticity describes what happens when individual neurons, or small groups of them, encode more distinct features or concepts than there are neurons to go around. Anthropic's work on superposition showed that large transformers pull this off by representing features as linear combinations of activations, cramming exponentially more concepts into a fixed-size space.
The tradeoff is that individual neurons stop meaning one thing. Sparse autoencoders (SAEs) are the standard fix: they learn a much wider, sparse basis where each direction ideally corresponds to one interpretable concept. Even after SAE decomposition, the authors show, two features that look totally unrelated in meaning can still share activation geometry inside the model. Enhance one, and the other moves with it.
Four ways to poke the model
The researchers mapped polysemantic interference in Pythia-70M and GPT-2-Small using pretrained SAEs from Neuronpedia. They define the interference scale between two SAE features as the cosine similarity between their activation-space directions, and separately measure surface semantic similarity using embeddings of glosses generated by GPT-4o-mini. Feature pairs with high interference but low semantic similarity are the interesting ones.