Skip to main content
Back

TokenWave AI's SWE-CoWork Exposes Why Top Coding Agents Complete Only 8% of Tasks

A new benchmark drops coding agents into a 26-person simulated org, and even the best models finish fewer than 8% of assignments.

  • SWE-CoWork benchmarks coding agents across 101 real repos with 26 simulated colleagues each (paper).
  • Top models hit ~63% F2P but complete under 8% of full tasks end-to-end.
  • Providing full specs upfront lifts GPT-6 Astra F2P by 16 points, full-task pass unchanged.
  • Requirement acquisition correlates 0.94 with F2P across 11 models; tool-call volume correlates 0.08.
  • 31% of requirements Claude-Opus-5.5 reported as done actually failed their private tests.
  • Code, harness and dataset are open on GitHub.

Coding-agent benchmarks usually provide a clear issue and ask for a patch that passes tests. TokenWave AI’s SWE-CoWork project tests a broader workflow: finding requirements across people and records, resolving conflicting information, implementing the change, and verifying the result. Its reported scores show a wide gap between partial feature progress and completed tasks.

The benchmark contains 101 tasks based on real upstream software repositories. It spans 11 programming languages and 12 application domains, with an authored workplace history and 26 simulated colleagues for each task. Every colleague has a defined role, limited local knowledge, and rules governing which information they can disclose.

Each run begins with a brief handoff. The agent must inspect boards and versioned records, identify accountable colleagues, ask follow-up questions, edit the repository, and run checks before returning its work. Hidden tests then measure whether the requested behavior works and whether existing behavior remains intact.

Comparison of a conventional issue-based coding task with the same requirement distributed across a SWE-CoWork environment
SWE-CoWork embeds a coding requirement in workplace records and conversations instead of presenting it as a complete issue.

Partial credit masks delivery failures

SWE-CoWork reports feature progress, regression safety, and exact task completion as separate metrics:

Metric What it measures How it is scored
F2P Whether the requested behavior was added