Skip to main content
Back

CodeSpecBench Reveals Top AI Models Fail to Describe What Code Actually Does

A new benchmark reveals frontier LLMs can write code but struggle to formally describe what that code should actually do.

  • CodeSpecBench evaluates LLMs on generating executable Python preconditions and postconditions from natural language, testing semantic understanding beyond code generation.
  • Best model Claude-4.5-Sonnet reaches only 20.2% pass rate on repository-level tasks from SWE-bench Verified, exposing weak program semantics grasp.
  • On function-level LeetCode tasks, top scores are GPT-5-mini at 47.0% and Gemini-2.5-Pro at 46.2%.
  • Specifications are significantly harder than solutions: Claude drops from 77.2% solution pass to 21.0% spec pass rate.
  • Function-level errors dominated by type and range violations (48-54%); repo-level by dependency resolution and over-restrictive asserts.
  • Benchmark, data, and evaluation harness are open source at SparksofAGI/CodeSpecBench.

Writing code that passes tests is one thing. Writing down, precisely and executably, what a piece of code is supposed to do is much harder, even for the strongest models available today. A new benchmark called CodeSpecBench puts 15 state-of-the-art LLMs through this exact test, and the results suggest that fluency in code generation does not translate into a real grasp of program semantics.

Writing code vs. knowing what it means

When an LLM produces code that passes tests, does it actually understand the intended behavior, or is it pattern-matching against similar solutions it has seen? Executable behavioral specifications, defined via preconditions and postconditions, offer a concrete way to probe that question.

Preconditions describe what inputs and program state a function is allowed to receive. Postconditions describe what must be true after the function runs. Together they form an executable contract, a machine-checkable description of intent that lives separately from the implementation. Preconditions typically constrain input types and value ranges, and may also encode assumptions about program state, such as object fields, global variables, configuration settings, or external resources.

Prior benchmarks in this space had significant blind spots. FormalBench and VERINA both rely on deductive verification and exclude repository-level tasks. FormalBench depends on the OpenJML verifier, whose results can be unstable under missing invariants and complex control flow, and its specifications are derived from code rather than natural-language descriptions. Others were limited to toy problems or ignored preconditions entirely.

What CodeSpecBench actually measures

The benchmark reframes specification generation as an execution problem rather than a formal-verification problem. Models are asked to translate a natural-language problem description into two Python functions, preconditions and postconditions, using only assert statements. Those functions are then executed against curated test suites.

CodeSpecBench pipeline diagram showing specification generation, evaluation, and execution stages

Two evaluation dimensions matter:

  • Correctness: does the specification accept all valid behaviors? A spec that rejects a legitimate test case is too tight.
  • Completeness: does it reject invalid behaviors? A spec that accepts everything is trivially true and useless.