Skip to main content
Back

Layer 6 AI's AMC Beats GRPO Without Ever Touching Model Weights

Agentic Monte Carlo brings RL-style optimization to closed APIs like GPT-4o and Claude without touching model weights, beating GRPO at scale.

  • Core idea: Agentic Monte Carlo (AMC) applies Sequential Monte Carlo to optimize black-box LLM agents at inference time, no weight updates needed.
  • Key result: AMC outperforms GRPO -- a full RL fine-tuning method -- as test-time compute scales, while never touching model parameters.
  • How it works: Runs N parallel agent trajectories as "particles," uses a learned value function to resample promising ones mid-task, steering the agent toward success.
  • What gets trained: Only a lightweight value function on interaction histories; the underlying black-box LLM (e.g. GPT-4o, Claude) is untouched.
  • Benchmarks: Validated on WebShop, SciWorld, and TextCraft from the AgentGym suite; consistently beats prompting baselines across all three.
  • Code available: Open-sourced at github.com/layer6ai-labs/Agentic-Monte-Carlo.

Reinforcement learning has become the go-to recipe for making LLM agents smarter. But there is a catch: it only works if you can actually update the model's weights. LLM agents operate in two distinct regimes -- open-weight agents amenable to reinforcement learning, and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. In other words, if you are building on top of GPT-4o, Claude, or Gemini, you are locked out of the most powerful training techniques available.

A new paper from Layer 6 AI proposes a principled workaround. They propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is treated as a posterior over trajectories, with the fixed black-box LLM agent serving as the prior. They employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged.

The Wall Every API Builder Hits

The dominant RL algorithms for agents -- PPO and more recently GRPO -- require gradient access to the model being trained. RL has become the dominant training paradigm through methods like PPO and GRPO, and these algorithms have proven highly effective for open-weight models, enabling abilities ranging from complex mathematical reasoning to software engineering. But such parameter-updating approaches result in prohibitive computational cost, which typically restricts application to LLMs with fewer than 32B parameters, and they only achieve diminishing returns compared to simply using larger, more powerful frozen LLMs.

The practical consequence is stark: teams building agentic products on top of frontier proprietary models have no principled way to optimize their agent's behavior beyond prompt engineering. Best-of-N sampling -- running the agent multiple times and picking the best result -- is the closest thing to a baseline, but it is naive: every trajectory is generated independently, with no mechanism to steer the agent toward better outcomes mid-task.