Rep2Text Recovers Private Prompts From a Single AI Hidden Vector
A new framework called Rep2Text shows that a single hidden vector inside an LLM leaks about half the words of its input, raising privacy and interpretability questions.
- Rep2Text reconstructs text from a single last-token LLM representation using a trainable adapter plus a decoder LLM. Paper
- Recovers roughly half of tokens in 16-token Wikipedia inputs while keeping topic and structure intact.
- Tested on Llama-3.1-8B, Llama-3.2-3B, Mistral-7B-v0.1, Gemma-7B targets with Llama and Qwen decoders up to 32B.
- Cross-model decoding works: attacker does not need the same model family as the target.
- Beats Vec2Text decisively on out-of-distribution clinical notes, suggesting a general representation-to-text mapping.
- Decoder scale barely helps; recoverability is bounded by what the target representation itself encodes.
Researchers have shown that a single activation from a large language model can reveal substantial parts of the text that produced it. Rep2Text takes the hidden representation of a prompt’s final token, projects it through a trained adapter, and lets a second LLM reconstruct the input autoregressively. On held-out 16-token passages, the system reaches 0.45 to 0.52 ROUGE-1 and often preserves the original topic, entities, and sentence structure.
The result clarifies how much input information survives inside the compressed state used for next-token prediction. It also exposes a security risk for systems that transmit, cache, or share intermediate activations. Exact wording remains difficult to recover, especially for longer inputs, but one vector can still disclose meaning and structure.
- Input: One hidden vector from the final prompt token at a selected transformer layer.
- Attack: A trained adapter maps that vector into pseudo-token embeddings for a frozen decoder.
- Best results: Short inputs drawn from distributions similar to the training data.
- Primary risk: Topic, structure, and some wording can leak even when raw prompts are unavailable.
How much survives the bottleneck?
A transformer’s final-token representation summarizes the preceding context for next-token prediction. The model has no explicit reconstruction objective, so the unresolved question was how much of the original text remained accessible in that summary. Earlier inversion methods generally targeted sentence embeddings from encoder models such as BERT, consumed full hidden-state sequences, or relied on repeated optimization.
Rep2Text tests a narrower threat model: an attacker obtains one vector h from a known layer and trains a model to infer the source text. The reconstruction uses no iterative search, partial text span, or task-specific hint at inference time. Training still requires paired examples of source text and target-model activations, and the adapter is tied to the representation space it learned.
An activation becomes pseudo-tokens
Rep2Text follows the projection pattern used by vision-language systems such as LLaVA. A two-layer multilayer perceptron with a gated skip connection maps the target model’s activation into the decoder’s token-embedding space. The adapter reshapes its output into
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
