Tsinghua's MARS² Beats O4-Mini on Coding by Training AI Teams Together
A new RL framework has heterogeneous agents collaborate inside a shared search tree, lifting Qwen3-8B by 8 points on LiveCodeBench.
PRO- MARS² trains multiple agents together inside a shared search tree, treating the tree as the RL environment.
- Qwen3-8B gains 8.0 points Pass@1 over base and 4.4 over vanilla GRPO on LiveCodeBench v6.
- AReaL-14B reaches 64.6% Pass@1, edging out O4-Mini (Low) at 63.7%.
- Method combines Thompson-sampled node assignment, tree-consistent reward shaping, and path-level group advantage.
- Extended variant with two 32B agents hits 77.7%, reportedly beating GPT-5.1 on the same benchmark.
- Code available at github.com/TsinghuaC3I/MARTI as part of the MARTI-v2 release.
Reinforcement learning has become the default recipe for pushing large language models past their supervised ceilings on code and math tasks. There is a stubborn wall, though: once a single policy has explored the trajectories it can easily reach, returns diminish quickly. A team from Tsinghua's C3I lab proposes a way through by making several independently trained agents share one search tree and training them together.
The framework, called MARS² (Multi-Agent Reinforced Tree-Search Scaling), reframes the search tree itself as the RL environment. Independently optimized agents collaborate inside a shared tree-structured search space, modeled as a learnable multi-agent interaction environment, so heterogeneous models can jointly generate and refine candidate solutions within a common search topology.
Where single-policy RL runs out of runway
Group Relative Policy Optimization (GRPO), the technique behind DeepSeek-R1 and similar recipes, samples a batch of rollouts per prompt, scores them, and pushes the policy toward the better ones. It works, but the batch is drawn from one policy, so the exploration signal narrows over time. Search-enhanced RL adds structured exploration yet stays constrained by single-agent priors. Multiple interacting policies can supply more diverse exploratory signals, but existing approaches typically sit outside any structured search.
Tree search adds structure but keeps one brain. Multi-agent debate adds brains but throws away structure. MARS² aims for both.
How the tree becomes a training environment
The framework has three moving parts working in a loop:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
