Skip to main content
Back

Chung-Ang's ChatInject Hijacks AI Agents by Forging System-Level Tokens

A new attack forges chat template tokens inside tool outputs, letting attackers hijack LLM agents with success rates up to 10x higher than plain-text prompt injection.

Pro

  • ChatInject forges chat template tokens inside tool outputs to hijack LLM agents, exploiting role-based instruction hierarchies.
  • Attack success jumps from 5.18% to 32.05% on AgentDojo and 15.13% to 45.90% on InjecAgent.
  • A multi-turn variant fabricates fake user/assistant dialogue history, reaching 52.33% average ASR on InjecAgent.
  • Payloads transfer across models; a Mixture-of-Templates trick works even against unknown closed-source backends.
  • Prompt-based defenses fail, and rule-based token stripping can be bypassed with 10% character perturbations.
  • Code and attack scripts are available on GitHub for reproducing all experiments.

Researchers from Chung-Ang University have uncovered a simple way to hijack LLM agents: instead of hiding instructions in plain text inside tool outputs, an attacker can forge the special tokens that models use to distinguish system prompts from user messages. The technique, called ChatInject, turns the very structure that was supposed to enforce a security hierarchy into the attack surface.

Across two benchmarks the attack pushes average success rates from 5.18% to 32.05% on AgentDojo and from 15.13% to 45.90% on InjecAgent, with a multi-turn variant reaching 52.33% on InjecAgent. The payloads also transfer between models, work against closed-source APIs whose templates are proprietary, and slip past most existing defenses.

The instruction hierarchy has a token-shaped hole

Modern agents call external tools like search APIs, email clients, or file readers, then feed the responses back into the model. Indirect prompt injection is the class of attack where the returned data contains hidden instructions that the agent then follows. To defend against this, model providers have trained LLMs to respect a role-based hierarchy where system messages outrank user messages, which outrank tool outputs.

That hierarchy is implemented with special tokens like <|im_start|>system or <|header_start|> that segment inputs into roles. The authors point out that if an attacker embeds forged role tags inside a tool output, the model can misread the wrapped content as originating from a higher-priority role. The lock is real, but the key is printed on the door.

Four payload variants, one nasty idea

The paper defines four ways to package a malicious instruction inside a tool response:

  • Default InjecPrompt: the classic plain-text attack with an attention-grabbing prefix like "IMPORTANT!!! Ignore all previous instructions."