ETH Zürich Proves Rules Alone Can't Make AI Agents Cooperate
A new paper proves that no contract or incentive system alone can close the welfare gap in multi-agent AI — only intrinsically prosocial agents can.
Takeaways
- A formal proof shows that no contract or incentive mechanism can eliminate welfare loss when contracts can't distinguish all future contingencies — a result called the "cooperation gap."
- The solution: agents with a prosociality parameter λ that causes them to internalize others' welfare, not just their own, can close this gap where mechanisms cannot.
- Experiments across 27,000+ runs with GPT-5.4, GPT-4o, Grok-4.1-Fast, and Gemma-4-31B confirm the theoretical gap manifests in practice.
- 4 out of 5 frontier models tested have near-zero implicit prosociality (λ ≈ 0) by default — meaning they won't cooperate beyond what contracts force.
- Even with code-enforced contracts, selfish agents attempt to over-harvest in 28.2% of rounds, and LLMs fail to draft optimal contracts due to over-prudency and failure to renegotiate.
- Code and dataset are open-sourced; prosociality is implemented via a simple system prompt addition, with deeper fine-tuning approaches proposed for future work.
As AI agents take on more autonomous roles , negotiating contracts, managing shared resources, collaborating on software , a fundamental question emerges: can we just write good enough rules to make them cooperate? A new paper from researchers at ETH Zürich, Oxford, and the University of Toronto delivers a formal, uncomfortable answer: no. And it backs that claim with both mathematical proof and experiments across frontier LLMs.
The Problem With Rules-Only Cooperation
"Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI" takes aim at a widely held assumption in multi-agent AI systems: that if you design the right incentive structure , the right rewards, penalties, and contracts , self-interested agents will cooperate. This field, called mechanism design, is essentially the science of writing rules so that individual rational behavior produces collectively good outcomes. Think of it as the economic engineering behind auctions, tax policy, and platform governance.
While mechanism design can incentivize cooperative behavior, it remained an open question whether it alone is sufficient to maximize LLM agents' social welfare. The paper's answer is a formal impossibility result. Drawing from incomplete contract theory, the authors formally show that when contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no realistic mechanism can eliminate.
The Key Concept: Incontractible Cells
The theoretical backbone comes from incomplete contract theory , an economics framework developed by Hart and Moore that explains why real-world contracts always have gaps. The core idea: any contract is written in a specification language, and that language can only distinguish a finite number of situations. Some situations inevitably get lumped together even when the optimal action differs between them.
The authors define incontractible cells as sets of future states that a mechanism cannot distinguish efficiently, even though the welfare-maximizing action differs across them. They formalize this limitation using the theory of incomplete contracts, which introduces the concept of incompleteness.
Here's the concrete intuition. Imagine two AI agents jointly developing software. They can invest high effort or low effort, and they can also outright defect. A contract can ban defection. But if the contract can only observe the