Prime Intellect's prime-rl Ships Six RL Algorithms With Per-Environment Control

prime-rl, Prime Intellect's large-scale RL training framework, just got a major architectural upgrade: a first-class algorithms layer that ships six built-in training algorithms and makes it trivially easy to bring your own. The change is live on main today, and it quietly solves one of the most annoying problems in multi-environment RL training.

The problem with bolted-on algorithms

Before this update, prime-rl was primarily designed as a highly performant framework for large-scale async RL. Other algorithms were added over time, but they were not accounted for in the original design. Each new algorithm arrived as a thread through the entire system rather than as a module. In practice, that meant a top-level training_mode switch branching the orchestrator, loss selection buried inside the trainer, and only one loss function allowed per batch. Adding a new algorithm meant touching internals across the whole stack.

The goal of introducing an algorithms layer was to centralize everything algorithm-specific in one place and give it a real abstraction , a space researchers can hack on without giving up on performance or having to touch trainer internals.

Six algorithms, one abstraction

The six algorithms that ship built-in cover a wide spectrum of what the post-training field has converged on:

• GRPO (default) , the standard group-relative policy optimization. Samples a group of rollouts per prompt, normalizes rewards across the group, and uses that as the advantage signal. No critic network needed.
• MaxRL , a recent CMU paper that argues standard RL only optimizes a first-order approximation of the true likelihood over correct rollouts. MaxRL Pareto-dominates existing methods in all models and tasks tested, achieving up to 20x test-time scaling efficiency gains compared to its GRPO-trained counterpart.
• OPD (On-Policy Distillation) , the student generates its own rollouts on-policy, while a frozen external teacher model provides token-level supervision via reverse KL divergence. This avoids the distribution mismatch of standard SFT.