Peking University's CodeFlowBench Cuts Frontier AI Coding Accuracy in Half
A new benchmark decomposes Codeforces and GitHub problems into dependency chains, exposing sharp performance drops when LLMs must build code iteratively.
Takeaways
- CodeFlowBench is the first benchmark to test multi-turn, dependency-aware code generation where LLMs must reuse their own previously written functions. Paper
- Two datasets: 5,258 Codeforces problems (Comp) plus GitHub-sourced repos across five domains (Repo), decomposed automatically via AST parsing.
- Most models score below 30% Pass@1 in multi-turn mode, with reasoning models like DeepSeek-R1 losing over 50% versus their single-turn scores.
- Gemini-3-flash-thinking leads Comp at 48.4%; GPT-5 leads Repo at 34.7%; all models collapse when Dependency Structure Complexity exceeds 1.33.
- Three new metrics (APT, APD, DSC) capture partial progress and dependency complexity rather than binary pass/fail.
- Code, data, and harness are open source at the official repo; accepted to ACL 2026.
A team from Peking University has built the first benchmark that measures how well large language models can write code the way real engineers do: one function at a time, reusing what they wrote before. The results are unflattering. CodeFlowBench shows that even frontier models lose roughly half of their coding accuracy when forced to build a solution incrementally rather than dumping it all in one shot.
The gap nobody was measuring
Most code benchmarks treat generation as a single-shot exercise. You give the model a prompt, it emits a function, you run the tests. HumanEval, MBPP, and even SWE-Bench focus on standalone functions or single-turn code modification, leaving multi-turn iterative generation largely unmeasured. Production software is written differently. Developers decompose a task into helpers, implement the leaves of a dependency tree, then compose them upward into a top-level solution.
The authors call this workflow codeflow, and argue that no existing benchmark actually tests it. Earlier multi-turn efforts like MTPB only exercised single-function programming without unit tests or realistic dependencies. CodeFlowBench fills that gap by forcing models to build modular solutions across many turns, using previously implemented functions as building blocks.
Two datasets, one pipeline
CodeFlowBench comprises two complementary components: CodeFlowBench-Comp, a core collection of 5,000+ competitive programming problems from Codeforces updated via an automated pipeline, and CodeFlowBench-Repo, which is sourced from GitHub repositories to better reflect real-world scenarios. Comp targets algorithmic reasoning depth. Repo targets domain breadth across five specialized areas: Basic, Cryptography, Network, System, and Visualization.