Skip to main content
Back

UC Davis Puts LLaMA 2 Inside a Fuzzer, Finding 47 Bugs

Fine-tuned LLaMA 2 mutates structured inputs like PNG, XML, and JSON, uncovering 47 unique bugs and 27% more branch coverage than AFL++.

Pro

  • LLAMAFUZZ fine-tunes LLaMA 2-7B to mutate structured inputs like PNG, XML, JSON, PDF, and ELF while preserving format validity.
  • Found 47 unique bugs on the Magma benchmark, beating AFL++, Moptafl, Honggfuzz, and Fairfuzz.
  • Achieved 27.19% more branch coverage than AFL++ on real-world OSS-Fuzz programs across 15 targets.
  • Asynchronous dual-queue design lets GPU-bound LLM mutations run without blocking the fast AFL++ fuzzing loop.
  • Trained via LoRA on ~10,000 seed pairs per target in about 6 hours on a single A100 GPU.
  • Code, scripts, and checkpoints are open source at github.com/SecurityLab-UCD/LLAMAFUZZ.

Greybox fuzzers like AFL++ have found countless bugs by mutating program inputs at the bit level and watching what crashes. That strategy hits a wall when the input has to be a valid PNG, PDF, or XML document. Random bit flips almost always corrupt the header, the CRC, or the chunk structure, and the target program rejects the input before any interesting code runs. A team from UC Davis and the University of Hong Kong wants to fix that by putting a fine-tuned LLaMA 2 inside the fuzzing loop. Their system, LLAMAFUZZ, learns what structured inputs look like and mutates them without breaking the format.

Why bit flips fail on structured data

The paper frames this as a long-standing trade-off. General purpose fuzzers are fast and universal but blind to structure, while grammar-based tools like Gramatron, Nautilus, and GRIMOIRE produce valid inputs but need hand-written grammars in specific normal forms. Relying on those rules for seed generation requires extensive domain knowledge, which limits adoption. The authors want the coverage-guided efficiency of AFL++ paired with the structural awareness of a grammar fuzzer, without writing any grammars.

Their example makes the problem concrete. Given a valid XML document, AFL++ tends to produce unclosed tags or garbled bytes. LLAMAFUZZ instead generates mutations that keep the XML declaration intact, invent new message elements, and swap child text values. On a PNG file, AFL++ frequently corrupts the 8-byte signature or the CRC-protected chunks, so libpng rejects the input immediately. The LLM-mutated version keeps the signature and chunk boundaries but changes semantic fields like the IHDR filter method.