Prime Intellect's prime-rl Ships Six RL Algorithms With Per-Environment Control

prime-rl, Prime Intellect's large-scale RL training framework, just got a major architectural upgrade: a first-class algorithms layer that ships six built-in training algorithms and makes it trivially easy to bring your own. The change is live on main today, and it quietly solves one of the most annoying problems in multi-environment RL training.

The problem with bolted-on algorithms

Before this update, prime-rl was primarily designed as a highly performant framework for large-scale async RL. Other algorithms were added over time, but they were not accounted for in the original design. Each new algorithm arrived as a thread through the entire system rather than as a module. In practice, that meant a top-level training_mode switch branching the orchestrator, loss selection buried inside the trainer, and only one loss function allowed per batch. Adding a new algorithm meant touching internals across the whole stack.

The goal of introducing an algorithms layer was to centralize everything algorithm-specific in one place and give it a real abstraction , a space researchers can hack on without giving up on performance or having to touch trainer internals.

Six algorithms, one abstraction

The six algorithms that ship built-in cover a wide spectrum of what the post-training field has converged on:

• GRPO (default) , the standard group-relative policy optimization. Samples a group of rollouts per prompt, normalizes rewards across the group, and uses that as the advantage signal. No critic network needed.
• MaxRL , a recent CMU paper that argues standard RL only optimizes a first-order approximation of the true likelihood over correct rollouts. MaxRL Pareto-dominates existing methods in all models and tasks tested, achieving up to 20x test-time scaling efficiency gains compared to its GRPO-trained counterpart.
• OPD (On-Policy Distillation) , the student generates its own rollouts on-policy, while a frozen external teacher model provides token-level supervision via reverse KL divergence. This avoids the distribution mismatch of standard SFT.
• OPSD (On-Policy Self-Distillation) , a learning algorithm where a single LLM acts as both teacher and student with different contexts, using a demonstration-conditioned version of the live policy as its own teacher. No separate model required.
• SFT , supervised fine-tuning on a frozen teacher's tokens, expressed as a cross-entropy loss. Unified under the same abstraction as the RL methods.
• ECHO , the most novel of the six. Combines GRPO on the model's own action tokens with a cross-entropy loss on environment-provided observation tokens (tool outputs, terminal responses). The model learns to predict what the environment will do in response to its actions , world modeling for free, at no extra forward pass cost.

ECHO: teaching models to model their world

ECHO deserves a closer look because it represents a genuinely new idea in LLM post-training. Standard RL only trains on the tokens the model itself generates, discarding the environment's responses as context. ECHO instead injects observation prediction directly into on-policy GRPO, requiring no separate corpus, world-modeling stage, feedback generator, dynamics model, or inference-time simulation.

In Prime Intellect's earlier experiments on ECHO, the results were promising. Training on a low-resource programming language called Forth, ECHO improved in-domain generalization without degrading performance on unrelated environments. The key insight is that ECHO works best when tool outputs are complex and predictable without memorization , like compiler output or code execution results , and degrades when applied to pure retrieval environments where predicting search results mostly just means memorizing documents.

The alpha parameter controls how much weight the SFT loss on environment tokens gets relative to the RL loss. Since each component is normalized by its own global token count, an ECHO environment that trains on far more tokens never dilutes the gradients of a GRPO environment packed beside it.

Per-environment algorithm selection

The headline architectural feature is that algorithms resolve per environment, not per run. This sounds like a small detail but it changes what's possible in a fundamental way.

Algorithms resolve per environment, not per run, because the right training signal is a property of the environment. ECHO makes sense for a terminal coding environment where tool outputs are deterministic and complex. It makes much less sense for a web search environment where predicting retrieved documents mostly leads to memorization. Forcing one algorithm across a mixed-environment run means picking the wrong signal for some environments.

The TOML config makes this concrete:

[orchestrator.algo]
type = "grpo"
[[orchestrator.train.env]]
id = "math-env"    # inherits grpo
[[orchestrator.train.env]]
id = "search-env"  # inherits grpo
[[orchestrator.train.env]]
id = "terminal-env"
algo = { type = "echo" }

A less obvious benefit falls out for free: multi-teacher OPD , each environment declares its own algorithm, so each environment can distill from its own teacher , a code-specialized teacher on your coding environment, a math-specialized one on your math environment, in a single run against a single student. To their knowledge, no other open-source RL framework lets you mix training algorithms per environment like this.

The abstraction underneath

Every algorithm is one file under orchestrator/algo/. The base class fixes three things: which model generates rollouts (the live policy or a frozen endpoint), how credit is assigned per token, and which of three fixed loss types consumes that credit (rl, ce, or ref_kl). In code:

class Algorithm:
    action_loss_type: ClassVar[ActionLossType] = "rl"  # rl | ce | ref_kl
    async def setup(self) -> None: ...          # connect frozen endpoints
    async def score_rollout(self, rollout: Rollout) -> None: ...   # per rollout
    async def score_group(self, group: list[Rollout]) -> None: ... # full group

The trainer knows exactly three loss components , rl, ce, ref_kl , and everything an algorithm decides reaches it as per-token weight streams: data, not code. Samples from different algorithms pack into the same micro-batch, because to the trainer they are indistinguishable. This is what makes mixed-algorithm batches work without any special-casing in the training loop.

What's still missing

The team is explicit that this is the first of several axes they want to open up. The roadmap includes:

• Smarter sampling , online difficulty filtering, curriculum scheduling, and replay buffers. The Sampler currently only decides which model generates rollouts; soon it will decide which examples get sampled at all.
• Agentic judging , judges that are themselves agents with tools and multi-turn access to the full rollout trace, rather than single-turn pattern matchers.
• Agentic credit assignment , a model that reads the trace and assigns different weights to different parts of the same rollout.
• Flexible dependency barriers , today the group is the only synchronization primitive. The team wants algorithms to declare arbitrary dependency sets, so credit assignment can wait on whatever structure it actually needs.
• Multi-agent environments , proposer-solver setups and self-play, which the current single-agent environment design doesn't support.

Who this is for

prime-rl is a framework for large-scale reinforcement learning, designed to be easy to use and hackable, yet capable of scaling to 1000+ GPUs. The algorithms layer is the part that matters most if you're doing research: you can now write one file to define a new algorithm and run it at full scale without touching the trainer. If you'd rather skip the infrastructure entirely, Prime Intellect also offers hosted post-training through their Lab platform. The layer is on main today, free and open-source.