Backdoored LLMs Can Extract Private Fine-Tuning Prompts

Researchers at Tsinghua University’s Conversational AI group have demonstrated a supply-chain attack against open-weight language models. A model publisher can implant a backdoor before release, wait for another organization to fine-tune the checkpoint on proprietary prompt-response pairs, and then recover private prompts by querying the resulting model through an ordinary API. The paper reports controlled experiments and does not document attacks observed in deployed systems.

On downstream datasets containing 5,000 samples, the attack fully recovered up to 76.3% of fine-tuning queries without prior knowledge of the private data. Recovery reached 94.9% when the attacker knew which words the queries tended to begin with. “Fully recovered” means that the generated text matched the complete training query.

The threat model in plain terms

A successful attack depends on four conditions that place the model publisher in a privileged position:

• The attacker can train or modify the base checkpoint before distributing it.
• The downstream team fine-tunes that checkpoint while computing loss on user-query tokens.
• The attacker can send repeated requests to the fine-tuned model, usually through its API.
• The implanted trigger survives downstream training and continues to influence generation.

The attack therefore targets organizations that obtain a checkpoint from an untrusted source, specialize it on sensitive data, and expose the result through an accessible inference endpoint. The attacker needs neither the fine-tuned weights nor gradient access.

Why private prompts carry value

The extraction target is the user side of a fine-tuning dataset: support tickets, medical questions, legal drafts, agent instructions, financial requests, and other curated inputs. Such prompts may contain confidential information, reveal internal workflows, or represent expensive data collection and filtering work.