CMAT Fixes Multi-Agent AI Coordination by Making Agents Agree First
A new centralized multi-agent RL framework replaces sequential action generation with a shared latent consensus vector, removing order-dependent bias.
Takeaways
- CMAT replaces MAT's sequential action decoder with an iterative latent consensus vector shared across agents.
- All agents act simultaneously conditioned on the consensus, eliminating action-order sensitivity in Multi-Agent Transformers.
- Joint policy factorizes into per-agent probabilities, enabling optimization with standard single-agent PPO.
- Outperforms MAT, PMAT, Triple-BERT, HAPPO, MAPPO on StarCraft II, MuJoCo, and Google Research Football hard tasks.
- Two fine-tuning modes (Consensus Enhancement, Action Policy Enhancement) provide further gains post-training.
- Code available at github.com/RS2002/CMAT, built on the PKU-MARL Multi-Agent-Transformer repo.
Cooperative multi-agent reinforcement learning has a nagging problem: the moment you try to coordinate a team of agents with a single Transformer, the order in which they generate their actions starts to matter. A new paper introduces the Consensus Multi-Agent Transformer (CMAT), which sidesteps that issue by making all agents agree on a shared latent strategy first, then act simultaneously.
The result is a framework that behaves more like single-agent PPO than a multi-agent algorithm, with reported gains on StarCraft II, Multi-Agent MuJoCo, and Google Research Football benchmarks. Code is on GitHub, built as a drop-in extension to the PKU-MARL Multi-Agent Transformer repository.
Why sequential coordination breaks down
Standard cooperative MARL hits the curse of dimensionality fast: joint observation and action spaces typically grow exponentially with the number of agents. The dominant workaround has been Centralized Training Decentralized Execution (CTDE), where a shared critic sees everything during training but agents act independently at runtime. That limits how tightly they can actually cooperate.
The Multi-Agent Transformer (MAT) went further by treating cooperative MARL as a sequence modeling problem, using a Transformer decoder to generate agent actions one after another. The resulting policy, however, can be highly sensitive to the action-generation order, and jointly optimizing that order blows up the search space to n factorial.
The authors illustrate the failure with a two-agent game where joint actions (A,A) and (B,B) are both Nash equilibria but only (B,B) is globally optimal. Under MAT's sequential decoder, if Agent 1 picks B and Agent 2 responds with A, the large negative reward crushes the probability of B for Agent 1 unconditionally, dragging the policy toward the suboptimal equilibrium. Sequential updating methods generally guarantee convergence only to a Nash Equilibrium, and conventional MAT-based methods may land at a Pareto-suboptimal one.