Skip to main content
Back

Örebro Researchers Warn LLMs Write Vulnerable Code 40% of the Time

Researchers at Örebro University synthesized 102 studies on LLMs and code security, mapping generated bugs, detection accuracy, repair limits, and poisoning risks.

Pro

Paper, Full Access
  • Örebro University researchers synthesize 102 studies on LLMs across code generation, detection, repair, prompting, and poisoning.
  • The review defines 10 vulnerability categories in LLM-generated code, with injection and memory bugs most common.
  • LLMs often match static analyzers on recall but produce false-positive rates over 60% in many detection studies.
  • StarCoder2-7B drops from 68% F1 on BigVul to 3% on the harder PrimeVul benchmark.
  • Chain-of-thought and tool-augmented in-context prompting reliably outperform zero-shot for both detection and repair.
  • Poisoning attacks like CodeBreaker evade static analyzers 92% of the time and were accepted by 9 of 10 developers.

A pair of researchers at Örebro University have published one of the most exhaustive surveys yet on what happens when large language models touch source code from a security standpoint. The systematic literature review by Enna Bašić and Alberto Giaretta pulls together 102 primary studies to answer a deceptively simple question: when an LLM writes, reviews, or patches code, does it make software safer or more vulnerable?

The short answer is complicated. LLMs produce insecure code at rates that vary wildly by task and language, they can rival dedicated scanners at spotting bugs while drowning developers in false positives, and their training pipelines are a live attack surface. The review is the first to bundle secure code generation, vulnerability detection, vulnerability fixing, unified detect-and-fix pipelines, prompting strategies, and data poisoning into a single evidence base.

Why bother with another survey

Prior reviews touched pieces of this landscape but never the whole thing. Among earlier secondary studies, only Negri et al. describe how LLMs often generate code containing specific vulnerabilities, but they did not provide a systematic categorization. The Örebro work organizes those vulnerabilities into 10 distinct categories. The authors also argue that no previous survey investigated how prompting strategy affects code security tasks, or how data poisoning ripples through detection and repair.

The methodology follows established guidelines from Petersen et al. The authors retrieved 7,008 records from an automated search across IEEE Xplore, ACM Digital Library, ScienceDirect, SpringerLink, and USENIX, ending with 102 included studies after screening and snowballing. The corpus spans 2021 through early 2026, with the bulk of the work coming from the last two years.

Ten flavors of insecure output

For code generation, the review catalogs vulnerability types across 21 studies and maps them to the Common Weakness Enumeration taxonomy. The categories the authors define are:

  • Injection (SQL, XSS, OS command, regex)
  • Memory management (buffer overflow, use-after-free, null pointer dereference)
  • File management (path traversal, unrestricted uploads)
  • Deserialization of untrusted data
  • Sensitive data exposure
  • Authentication and authorization
  • Cryptography (broken algorithms, weak hashing)