---
title: "LogicEval Exposes Where LLMs Fail to Fix Real Security Bugs"
subtitle: "A new benchmark of 86 real-world CVEs exposes why automated program repair and LLM prompting still stumble on business-logic bugs."
url: "https://alphasignal.ai/news/logiceval-exposes-where-llms-fail-to-fix-real-security-bugs"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T23:27:40.421Z"
type: "Paper"
topics:
  - "Security"
  - "Development"
subtopics:
  - "Red Teaming"
  - "Code Generation"
upvotes: 2283
source_url: "https://arxiv.org/abs/2604.12994"
comments: 0
truncated: true
---

# LogicEval Exposes Where LLMs Fail to Fix Real Security Bugs

> A new benchmark of 86 real-world CVEs exposes why automated program repair and LLM prompting still stumble on business-logic bugs.

Automated program repair has quietly become one of the more useful applications of large language models, but almost all of that progress has been measured against memory corruption bugs like buffer overflows. A new benchmark called [LogicEval](https://arxiv.org/abs/2604.12994) targets that blind spot by focusing on logical vulnerabilities, the kind of bugs where code compiles cleanly, passes its tests, and still lets an attacker bypass authentication or leak data.

## Why logical bugs break existing repair tools

Logical vulnerabilities stem from flaws in program logic rather than memory safety, and they routinely lead to critical security failures. They are harder to catch because there is no dangling pointer or corrupted stack to point at. Memory-safety bugs usually manifest through invalid memory accesses, and dataflow vulnerabilities can often be expressed as propagation from sources to sinks. Logic bugs instead come from missing, misplaced, or incorrectly composed enforcement of application-specific security invariants.

That makes them a poor fit for pattern-based scanners. Existing automated program repair techniques focus mostly on memory corruption and struggle here because they have limited semantic understanding of the vulnerable code and its expected behavior. LLMs have shown promise because they can reason about intent, but until now there was no principled way to measure how much of that promise holds up on real CVEs.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software](https://arxiv.org/abs/2604.12994)
- [LogicEval (arXiv HTML v1)](https://arxiv.org/html/2604.12994v1)
- [LogicEval (ACL Anthology PDF)](https://aclanthology.org/2026.acl-long.2136.pdf)
- [Antaeus: Hunting Repository-Level Logic Vulnerabilities via Context-Grounded LLM Reasoning](https://arxiv.org/pdf/2607.01138)
- [LogicPatcher: Patching Logic Vulnerabilities for Web Applications](https://research.ibm.com/publications/patching-logic-vulnerabilities-for-web-applications-using-logicpatcher)