LogicEval Exposes Where LLMs Fail to Fix Real Security Bugs
A new benchmark of 86 real-world CVEs exposes why automated program repair and LLM prompting still stumble on business-logic bugs.
PRO- LogicEval is a new framework for testing automated repair on logical, not memory, vulnerabilities.
- Ships with LogicDS, the first dataset of 86 real CVEs targeting business-logic flaws.
- Tests vary source code scope, auxiliary info, and prompting strategy including zero-shot, few-shot, and chain-of-thought.
- Finds failures driven mainly by prompt sensitivity, lost code context, and bad patch localization.
- Also asks the model to explain its patch, catching fixes that only look syntactically correct.
- Code, dataset, and generated patches released publicly for reproducibility and new-model evaluation.
Automated program repair has quietly become one of the more useful applications of large language models, but almost all of that progress has been measured against memory corruption bugs like buffer overflows. A new benchmark called LogicEval targets that blind spot by focusing on logical vulnerabilities, the kind of bugs where code compiles cleanly, passes its tests, and still lets an attacker bypass authentication or leak data.
Why logical bugs break existing repair tools
Logical vulnerabilities stem from flaws in program logic rather than memory safety, and they routinely lead to critical security failures. They are harder to catch because there is no dangling pointer or corrupted stack to point at. Memory-safety bugs usually manifest through invalid memory accesses, and dataflow vulnerabilities can often be expressed as propagation from sources to sinks. Logic bugs instead come from missing, misplaced, or incorrectly composed enforcement of application-specific security invariants.
That makes them a poor fit for pattern-based scanners. Existing automated program repair techniques focus mostly on memory corruption and struggle here because they have limited semantic understanding of the vulnerable code and its expected behavior. LLMs have shown promise because they can reason about intent, but until now there was no principled way to measure how much of that promise holds up on real CVEs.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
