TU Munich's IHO Breaks Layered AI Safety Defenses Without Touching Model Weights
IHO uses a masked diffusion model trained via preference optimization to jailbreak any LLM without white-box access, transferring across models and defenses without retraining
PRO- IHO (Indirect Harm Optimization) is a new black-box LLM jailbreak attacker that requires only output access to the target model.
- It uses a masked diffusion language model trained via iterative preference optimization against a harmfulness judge.
- The same trained model works as both a strong adaptive attack on specific behaviors and an amortized policy that transfers to unseen models without retraining.
- IHO outperforms state-of-the-art methods against layered defenses including Circuit Breakers + auxiliary detectors without defense-specific adaptation.
- The paper positions IHO as an AutoAttack equivalent for LLMs -- a standardized, reliable evaluator for comparing defenses fairly.
- Code and models are openly released on GitHub and Hugging Face.
Measuring how safe a language model really is turns out to be surprisingly hard. If your attack is too weak, you declare victory on a defense that would crumble under a stronger probe. This is the core problem with LLM jailbreak evaluation today: there is no agreed-upon, reliable attack that everyone uses as a baseline. A new paper from TU Munich argues this gap is dangerous, and proposes a method called Indirect Harm Optimization (IHO) to fill it.
The benchmark gap that keeps defenses dishonest
In image classification, this problem was largely solved years ago. The research community lacked a systematic understanding of adversarial robustness, and a key challenge was that evaluation was often error-prone, leading to robustness overestimation. AutoAttack, an ensemble of white- and black-box attacks, was shown in a large-scale study to improve almost all robustness evaluations compared to the original publications, and became the de facto standard for comparing image classifier defenses.
No equivalent exists for LLMs. HarmBench is a standardized evaluation framework for LLM safety auditing, focused on benchmarking both the propensity of LLMs to emit harmful completions and the robustness of various automated red-teaming, jailbreak, and refusal-defense strategies -- but the attacks it bundles are inconsistent in quality, computational cost, and applicability. A defense that looks strong against one attack may be trivially broken by another. Without a single reliable attacker, defense comparisons are essentially apples-to-oranges.
The authors identify three properties that a standardized LLM jailbreak evaluator must have, and that no existing method jointly satisfies:
- Black-box compatible -- it cannot require access to model weights or gradients, since most deployed models are closed-source
- Applicable to arbitrary defense pipelines -- it must work even when a model is wrapped in additional classifiers, filters, or representation-level defenses
- Efficient -- it must not require thousands of queries per behavior, making it practical at scale
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
