Skip to main content
Back

AgentForge Beats GPT-4o Single-Agent Coding by 26 Points With Sandbox Execution

A five-agent pipeline that forces every generated patch through a Docker sandbox hits 40 percent on SWE-Bench Lite, more than doubling single-agent GPT-4o.

Pro

  • AgentForge hits 40.0% on SWE-Bench Lite, versus 14% for single-agent GPT-4o and 18.1% for SWE-agent.
  • Five specialized agents: Planner, Coder, Tester, Debugger, Critic, coordinated by an orchestrator.
  • Every patch must survive a Docker sandbox with 512 MB RAM, 0.5 CPU, no network.
  • Ablations show removing the Planner collapses performance back to single-agent baseline levels.
  • Averages 13.6K input and 5.1K output tokens per task, roughly $0.14 per task on GPT-4o.
  • Open source at github.com/raja21068/AutoCodeAI, with dual retrieval over ChromaDB.

Language models can write code that looks right and still be wrong. A new framework called AgentForge tries to close that gap by treating sandboxed execution as a mandatory checkpoint rather than an optional afterthought, and the numbers suggest the approach matters more than model size. AgentForge resolves 40.0% of issues on SWE-Bench Lite, outperforming single-agent baselines by 26 to 28 points.

The verification gap in agentic coding

Most existing coding agents share a quiet weakness: they trust the model to know when its own code works. Current frameworks do not enforce grounded execution, instead inferring outcomes or leaning on permissive environments rather than observing actual program behavior. Next-token likelihood is not the same signal as a test suite returning green.

The authors frame software engineering with LLMs as a Markov decision process where actions are patches and the environment is a real interpreter running real tests. Bug fixing requires a feedback loop of plan, implement, execute, test, and revise, driven by execution signals. Without that grounding, models cannot reliably verify correctness or detect regressions, and simulated execution introduces systematic errors that propagate through the pipeline.

Five agents, one sandbox

AgentForge splits the work across five role-specific LLM calls, each with a fixed contract:

  • Planner produces a structured JSON plan with an ordered list of steps assigned to specific agents.