---
title: "Hochschule Esslingen Hits a Hard 0.30 Accuracy Wall Running Vul-RAG Locally"
subtitle: "A new reproducibility study of Vul-RAG finds the framework works without proprietary APIs, but every open-weight model tested, including reasoning models, hits a hard ceiling of ~0.30 pairwise accuracy on vulnerability detection."
url: "https://alphasignal.ai/news/hochschule-esslingen-hits-a-hard-0-30-accuracy-wall-running-vul-rag-locally"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-03T00:00:00.000Z"
updated: "2026-06-11T02:43:13.886Z"
type: "Paper"
topics:
  - "Retrieval"
  - "Open Source"
subtopics:
  - "Rag"
upvotes: 1077
source_url: "https://arxiv.org/abs/2606.04739"
comments: 0
truncated: true
---

# Hochschule Esslingen Hits a Hard 0.30 Accuracy Wall Running Vul-RAG Locally

> A new reproducibility study of Vul-RAG finds the framework works without proprietary APIs, but every open-weight model tested, including reasoning models, hits a hard ceiling of ~0.30 pairwise accuracy on vulnerability detection.

Security teams have long hoped that LLMs could automate the tedious work of hunting software vulnerabilities. [Vul-RAG](https://arxiv.org/abs/2406.11147), a RAG-based vulnerability detection framework published last year, made a compelling case: by grounding LLMs with structured knowledge distilled from historical CVEs, it pushed pairwise accuracy well beyond what raw LLMs could achieve. The catch? Its strongest results relied on proprietary models like GPT-4o and Claude, making independent verification nearly impossible. A new study from researchers at Hochschule Esslingen sets out to fix that, and what they find is both reassuring and sobering.

## The reproducibility problem hiding in plain sight

For approaches relying on proprietary models and APIs, reproducibility and replicability remain largely unexplored, raising the question of whether reported results generalize or depend primarily on specific model choices. This is not a minor footnote. If a security tool only works when you pipe code through a closed commercial API, you cannot audit it, run it air-gapped, or trust that results will be consistent as the underlying model silently updates. The ML community has increasingly recognized this gap, but vulnerability detection, a domain with real security stakes, has been slow to catch up.

The original Vul-RAG paper had already shown that LLMs have severe limitations in distinguishing between vulnerable and similar-but-benign patched code, with baseline accuracy of only 0.06 to 0.14\. Vul-RAG addressed this by building a structured knowledge layer on top of raw LLM inference, and the gains were real. But reproducing those gains without GPT-4o access was an open question.

## How Vul-RAG actually works

Vul-RAG works in three phases: first, it constructs a vulnerability knowledge base by extracting multi-dimensional knowledge via LLMs from existing CVE instances; second, for a given code snippet, it retrieves relevant vulnerability knowledge from the knowledge base based on functional semantics; third, it leverages LLMs to check the vulnerability of the given code snippet by reasoning the presence of vulnerability causes and fixing solutions of the retrieved vulnerability knowledge.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models](https://arxiv.org/abs/2606.04739)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG (original paper)](https://arxiv.org/abs/2406.11147)
- [Revisiting Vul-RAG - Full HTML paper](https://arxiv.org/html/2606.04739)
- [Revisiting Vul-RAG GitHub Repository](https://github.com/hs-esslingen-it-security/revisiting-Vul-RAG)
- [Vul-RAG ACM Publication](https://dl.acm.org/doi/10.1145/3797277)
