USTC's Asuka-Bench Exposes How AI Coding Agents Fail Real Web Development

Most code generation benchmarks hand a model a perfectly written spec and score the output once. That is not how software gets built. Asuka-Bench, a new benchmark from researchers at the University of Science and Technology of China, flips that assumption on its head by testing whether code agents can build working web apps from vague, incomplete prompts -- and then fix them based on user feedback, round after round.

The benchmark gap nobody was measuring

Existing code-generation benchmarks score a single mapping from a complete prompt to a one-shot output. Real web development is different: users seldom write a full spec at the start, and many requirements only become clear once they look at an intermediate result and react to it. This is the core insight driving Asuka-Bench.

The broader benchmark ecosystem has been evolving, but mostly in one direction. Code generation evaluation has evolved from function-level tasks to repository-level challenges, with early benchmarks like HumanEval, MBPP, and APPS now largely saturated. SWE-bench has become the de facto evaluation standard for coding agents, but its limitations are increasingly well documented: overly detailed issue descriptions inflate resolution rates, and single-language bias limits generalizability. What none of these benchmarks capture is the iterative, conversational nature of real product development.

Dominant benchmarks like SWE-bench and HumanEval assume a complete, self-contained task description -- an assumption systematically violated in real IDE use where specifications emerge over multiple turns from underspecified signals. A model optimized under this assumption may perform well on benchmarks while failing at tolerating ambiguity, tracking evolving intent, and proactively surfacing missing constraints.