Johns Hopkins Proves Steering Attacks on LLMs Can't Be Replicated by Real Prompts
A new proof shows activation steering pushes LLMs into internal states no text prompt can ever reach, with major implications for AI safety evaluations
Takeaways
- Core finding: Activation steering almost surely pushes LLM activations into states unreachable by any real text prompt, proven formally using real-analyticity of transformers.
- Key implication: White-box steering vulnerabilities do NOT imply equivalent black-box prompt vulnerabilities -- the two threat models must be evaluated separately.
- Empirical validation: Tested on Llama-3.2-1B, Qwen-2.5-0.5B, and Gemma-3-1B; prompt inversion (SipIt) fails at the first token for all steered activations.
- ICL finding: Many-shot in-context learning (up to 64 shots) can jailbreak models but through a completely different internal mechanism -- activations diverge further from steered states as shot count increases.
- "Here" attack: A trivial suffix jailbreak matches steering's attack success rate on most models, showing black-box risks exist independently of white-box ones.
- Code available: Experiments are reproducible via the invertsteer GitHub repo; paper on arXiv.
Activation steering has become one of the most popular tools in the LLM interpretability and safety toolkit. By adding a carefully crafted vector directly to a model's internal activations at inference time, researchers can suppress refusal, inject personas, probe truthfulness, and study how alignment is encoded inside a model. The technique is elegant, lightweight, and surprisingly powerful. But a new paper from Johns Hopkins University argues that its power comes with a fundamental catch: the internal states it creates are, in a precise mathematical sense, impossible to reach through any real text prompt.
The paper, "Steered LLM Activations are Non-Surjective" by Aayush Mishra, Daniel Khashabi, and Anqi Liu, frames this as a surjectivity problem. For a fixed model, does every steered activation have a "pre-image" -- a real text prompt that would produce the same internal state through the model's normal forward pass? Their answer, proven formally and validated empirically, is almost surely no.
Why This Question Matters Now
Activation steering has moved well beyond an academic curiosity. It is now used to control LLM behavior by adding semantically meaningful vectors directly into a model's hidden states during inference. It is often framed as a precise, interpretable, and potentially safer alternative to fine-tuning. In safety research specifically, steering demonstrations carry significant weight: if you can steer a model into producing harmful outputs by nudging its activations, the common interpretation is that those harmful behaviors are latent and accessible.
Anthropic reports that Claude 4.5 produced near zero unsafe responses in standard safety tests, yet activation steering that suppresses evaluation-awareness increased unsafe behavior, with one trial observing an 8% misalignment rate under a particular steering vector. Results like this have pushed the field toward treating white-box interventions as first-class threat models. The new paper doesn't dispute that steering is dangerous in open-weight settings -- it disputes what those results tell us about prompt-based risks.
The Core Argument: Activation Space Has Holes
Here is the key insight. A language model maps discrete text prompts (sequences of tokens from a finite vocabulary) into a continuous, high-dimensional activation space. Because the vocabulary is finite and the context window is bounded, the set of activation states reachable from real prompts is