Bespoke Labs' MAC Bench Reveals 8-Agent AI Teams Barely Outperform Solo Workers
Bespoke Labs' new 12-task benchmark separates two things: what a team gains from extra agents and what it gains from actually coordinating. For frontier models today, coordination adds almost nothing.
- Bespoke Labs released MAC Bench: 12 long, team-dependent tasks from Mars missions to Dubai air traffic control.
- Eight-agent teams beat single agents, but the gain comes from extra workers. Coordination's net effect is -0.018.
- Opus 5.5 leads teams at 0.53 vs GPT-6.1 Sol's 0.41, and is the only model gaining from coordination.
- Opus listens (72% inbox reads), GPT-6.1 Sol broadcasts 3.4x more, and Gemini 4 Argon mostly works alone.
- Teams cost 8.8x more for +0.07 score and reach solo scores only 2.2x faster.
- Harness open-sourcing, conflicting-goal tasks, and larger mixed-model teams are planned. Task contributions welcome.
Bespoke Labs has released Multi Agent Coordination Bench (MAC Bench), a benchmark of 12 four-hour tasks in which agents share workspaces, exchange information, or control separate parts of an environment. Across Opus 5.5, GPT-6.1 Sol, and Gemini 4 Argon, eight-agent systems improved mainly because they supplied more workers. Communication contributed little on average.
The result matters for developers building agent swarms because a message layer can consume tokens, create merge conflicts, and blur ownership without raising task scores. MAC Bench separates the benefit of parallel work from the effect of coordination, giving teams a baseline against which to test orchestration systems.
A four-hour coordination test
Most agent benchmarks were written for one agent, so adding workers rarely forces meaningful coordination. MAC Bench gives every configuration the same four-hour wall-clock limit and includes tasks whose state, files, budgets, or controls can create dependencies between agents.
- Mars Return: Design a sample-return mission in SysML v2, a systems-engineering language, and write its flight software under a $1.6 billion hardware budget. The grader flies the design through 48 hidden simulated worlds.
- Dubai ATC: Staff 11 controller positions at Dubai International Airport through 12 hours of recorded traffic and weather.
- Holdout-game: Control individual survivors in a continuously running zombie game. The team must build a stronghold and withstand nightly hordes that grow from 60 to 210 zombies.
- Power Outage: Write controllers for 16 areas of a national grid and restore service after a blackout.
- Other tasks: Build a lossless compressor for 21 file types, predict RNA structures, run a drone-assisted cave rescue, and play strategy games including Mindustry and 0 A.D.
MAC Bench divides the tasks according to whether coordination is structurally required:
- Shared tasks, 9 of 12
- Every agent receives the same information and tools. One agent could complete the task in principle, while larger teams can divide the work or edit common artifacts.
- Distributed tasks
- Each agent sees part of the environment or controls its own units. Useful state must pass between teammates for the group to act on it.
Three conditions isolate the teamwork effect
Bespoke Labs evaluates each model under three conditions, allowing the benchmark to distinguish parallel capacity from communication:
- S1: One agent completes the task.
- S8: Eight agents work with access to a shared message board.
- S8∅: Eight workers contribute without communication.
The silent condition differs by task type. For shared tasks, S8∅ is the best result from eight independent solo runs, which measures parallel sampling followed by selection. For distributed tasks, the same eight-agent system runs with its message channels disabled.
def decompose(s1, s8_silent, s8_team):
worker_gain = s8_silent - s1
coordination_gain = s8_team - s8_silent
return worker_gain, coordination_gain
decompose(0.35, 0.44, 0.42) # (0.09, -0.02)Averaged across the three models, the single-agent score was 0.35, the silent eight-worker score was 0.44, and the communicating team score was 0.42. Relative to one agent, silent workers gained 0.087 and communicating teams gained 0.070. The resulting coordination effect was -0.018 and was not statistically significant. Deterministic graders hidden from the agents scored each run at five checkpoints.
Message-board habits split the models
Solo performance was similar for Opus 5.5 and GPT-6.1 Sol, with scores of 0.39 and 0.38. Their communicating teams diverged: Opus reached 0.53, while GPT-6.1 Sol reached 0.41. Opus was the only tested model with a positive coordination gain, at roughly +0.06, and the GPT-6.1 Sol team remained below its silent baseline at every checkpoint.
- Opus 5.5 reads before acting. Inbox reads accounted for 72% of its message-board calls. Its most common failure involved independently functional parts that did not integrate, appearing in 37% of runs.
- GPT-6.1 Sol broadcasts heavily. Each agent sent 3.4 times as many messages as an Opus agent, while reads represented 41% of its board calls. Role confusion appeared in 51% of runs.
- Gemini 4 Argon struggles to divide ownership. Teams failed to establish a stable work split in 9 of 16 eight-agent runs. Duplicated work appeared in 74% of runs.
Message volume had little relationship with task performance. Within runs of the same task, the rank correlation between messages per agent and final score was +0.12, with a confidence interval that included zero. Message-board activity consumed 22% of tool calls with two agents and 27% with eight.
Coordination gains depend on the task
| Task | Type | Single agent | Worker gain | Coordination gain |
|---|---|---|---|---|
| Holdout-game | Distributed | 0.13 | +0.24 | +0.05 |
| Mindustry | Distributed | 0.17 | +0.08 | +0.08 |
| SOTA Compressor | Shared | 0.57 | +0.07 | +0.05 |
| Mars Return | Shared | 0.21 | +0.17 | -0.12 |
| Cave Rescue | Shared | 0.42 | +0.12 | -0.12 |
| Power Outage | Distributed | 0.27 | +0.12 | -0.07 |
| Dubai ATC | Shared | 0.20 | -0.02 | 0.00 |
Distributed access can create information worth sharing, as the positive results for Holdout-game and Mindustry show. Power Outage still recorded a negative coordination gain, while the shared compressor task gained from communication, so task category alone did not determine the outcome.
Within Holdout-game, a teammate changed its behavior within five minutes for 48% of reported findings, compared with 31% in other tasks. At the end of one run, an Opus agent reconstructed a map containing 13.3 landmarks in areas it had never visited. With communication disabled, that figure fell to 0.1.
Shared files and budgets created a different constraint in Mars Return. Opus scored 0.57 with four agents and 0.33 with eight as additional workers edited the same files and drew from the same hardware budget.
Failures begin at the handoff
To classify coordination failures, Bespoke Labs used Claude Sonnet 5.5 to label 18 failure types across logs from 139 team runs, then used Claude Opus 5.5 to check each labeled episode against the transcript. Duplicated work appeared in 64% of runs, though the review did not identify it as a source of material damage. Unused messages and incompatible components produced the consequential failures.
- The Dina shooting: Survivors questioned a stranger named Dina, quarantined her, and admitted her after her account of a mill pond matched their observations. One agent later interpreted a phrase she used as a thief's tell. Four teammates agreed without accounting for the earlier evidence, and the group shot her. The game reported that she had been honest, and the shooter acknowledged overriding “the real evidence.”
- A runway closed by keyword: In Dubai ATC, an agent wrote a bot that monitored the message board for “poor braking.” A teammate used those words in a weather note, causing the bot to close a runway for an unnecessary inspection. The closure produced two go-arounds and two fuel diversions.
- A near collision: Three Opus agents issued commands to the same aircraft. Conflicting climb instructions placed it at the same altitude as another plane and triggered a collision-avoidance alert in the simulator.
Eight agents cost 8.8 times more
At list prices, an eight-agent team cost about nine times as much as one agent while raising the average score from 0.35 to 0.42. A single agent delivered the highest score per dollar in all 36 task-model pairs, covering 12 tasks and three models. Mars Return cost between $12 and $89 for a single run and between $500 and $750 for an eight-agent run.
Teams reduced elapsed time in many cases, though their speedup remained far below linear scaling. In 30 of the 36 pairs, a team reached the solo agent's final score within the four-hour limit, with a median speedup of 2.2 times. Median time to that score was 84 minutes for Opus, 156 minutes for GPT-6.1 Sol, and 109 minutes for Gemini. Perfect eight-way scaling over a four-hour run would take 30 minutes.
Parallel first, coordinate selectively
Research labs are treating agent count as another way to spend inference-time compute. A test-time scaling study explores this approach, while Anthropic reported that early work on its Fermat formalization project failed when agents lost track of project state. Earlier benchmarks such as TeamBench found similar problems on shorter tasks, including coding workloads where subagents spent more tokens coordinating than implementing.
MAC Bench extends that work with hours-long runs and a silent-team baseline that estimates the contribution of communication. For systems resembling these tasks, its results support several engineering choices:
- Run independent agents in parallel when a trusted grader or selector can rank their outputs.
- Add communication when agents hold private state or control separate subsystems.
- Assign explicit ownership for shared files, budgets, resources, and external controls.
- Measure every orchestrated team against a silent or independent-worker baseline.
- Track whether messages are read and acted upon, rather than optimizing message volume.
Scope, access, and next steps
MAC Bench covers 12 tasks, three models, and a specific message-board design, so its measurements may change with other orchestrators, model mixes, or communication protocols. The aggregate coordination effect also carries statistical uncertainty. The published figures are Bespoke Labs' own benchmark results.
The public site currently provides a leaderboard, per-task pages, and a rollout viewer for inspecting transcripts. Bespoke Labs plans to open-source the harness, add tasks with conflicting agent goals, and test larger and mixed-model teams. Developers can propose tasks through its contact page.