Skip to main content
Back

UW-Madison Exposes How AI Hacking Agents Collapse Against Real Malware

A University of Wisconsin-Madison position paper maps six failure modes in LLM agents that reverse engineer binaries, from obfuscation blind spots to missing execution guardrails.

  • University of Wisconsin-Madison paper audits agentic reverse engineering systems and identifies six recurring failure modes.
  • Static agents fail on obfuscation: Control Flow Flattening drops decompilation success by over 70%.
  • Dynamic agents like CTF-Dojo and HackSynth accept arbitrary shell commands with no validation.
  • Adversarial binaries can weaponize LLM thinking latency by wrapping key logic in strict timeouts.
  • Hybrid agents risk infinite tool-call loops if obfuscation is designed to hinder agents rather than humans.
  • Read the full paper on arXiv.

Large language model agents are being pointed at some of the hardest jobs in offensive security, including tearing apart unknown binaries to figure out what they do. A new position paper from the University of Wisconsin-Madison takes stock of the field and argues that current agentic reverse engineering (RE) systems still collapse in the exact scenarios where they would matter most: obfuscated malware, timing-sensitive code, and unusual hardware architectures.

The authors, Salem Radey, Jack West, and Kassem Fawaz, audit the open-source codebases behind systems like HackSynth, PentestGPT, CTF-Dojo, and SWE-agent, then catalog six concrete failure modes and lay out research directions to close each one. Binary reverse engineering, the process of recovering a program's functionality from its compiled form, splits into static analysis, dynamic analysis, and hybrid approaches that combine the two.

Three flavors of agent, one messy problem

The survey sorts existing agents into three buckets by what they can actually touch. Static agents examine binaries without executing them, dynamic agents observe behavior at runtime, and hybrid agents mix both to capture more of a binary's behavior. The authors found no agent that does dynamic analysis in isolation; anything that runs a binary is doing hybrid work.