Prime Intellect Ships prime-rl to Train 744B Models on Just 28 GPUs

Prime Intellect's prime-rl v0.6.0 brings RL post-training to trillion-parameter MoE models with sub-5-minute step times on 28 H200 nodes

·
·
·
Prime Intellect Ships prime-rl to Train 744B Models on Just 28 GPUs
  • prime-rl v0.6.0 enables RL post-training of trillion-parameter MoE models with sub-5-minute step times on 28 H200 nodes.
  • Demonstrated on GLM-5 (744B MoE) running SWE agentic tasks at 131k context length with 256 rollouts per batch.
  • Router Replay (R3) captures expert routing decisions from inference and replays them on the trainer, cutting KL mismatch by ~10x.
  • P/D disaggregation + Mooncake KV cache prevents long prefills from stalling decode workers, enabling cross-replica prefix cache hits.
  • 3D parallelism (FSDP2 + Expert Parallelism + Context Parallelism) handles 131k sequences without blowing GPU memory budgets.
  • Fully open source under MIT/Apache 2.0; works with GLM-5, Kimi-K2, Nemotron-3, and other large MoE models.

Running reinforcement learning on a trillion-parameter model sounds like a recipe for GPU-melting latency and training instability. Prime Intellect just shipped prime-rl v0.6.0 to prove otherwise. The new release enables RL post-training of massive Mixture-of-Experts (MoE) models , think 744B+ parameter behemoths , on long-horizon agentic tasks, achieving sub-5-minute step times on only 28 H200 nodes. That's a meaningful systems milestone for anyone trying to post-train frontier-scale open models without a hyperscaler's budget.

Why RL at this scale is hard

In RL, inference is the critical bottleneck of the training lifecycle , that's where the model interacts with its environment, producing rollouts that are evaluated and assigned a reward. At trillion-parameter scale, the problem compounds: training such models introduces unprecedented challenges, including train-inference misalignment, inefficiencies in rollout processing, and bottlenecks in the RL system.

Agentic tasks make this even messier. A coding agent solving a SWE task might run for hundreds of turns, generating tool calls, reading file outputs, and reasoning over long contexts. Agentic tasks often have long-tail outliers , rollouts that can take up to a few hours, especially long-horizon coding tasks. Delaying the policy update until these rollouts finish would under-utilize GPUs and hurt performance.

The benchmark Prime Intellect set for themselves: train GLM-5 on SWE tasks at up to 131k sequence length, with sub-5-minute step times and a batch size of 256 rollouts, on only 28 H200 nodes. GLM-5.1 is Zhipu AI's flagship foundation model designed for autonomous agentic workflows, utilizing a massive 744 billion parameter MoE architecture, with 40 billion parameters active during inference.

Bar chart comparing trainer step time (3.7 min median) versus rollout step time (5.0 min median) for RL at 1T scale

Async-first, from the ground up

prime-rl is a framework for large-scale, asynchronous reinforcement learning of large language models, designed to be easy to use and hackable, yet capable of training 1T+-parameter MoE models on 1000+ GPU clusters. The key architectural decision is full asynchrony between the trainer and inference engine , they run on separate hardware and can be optimized independently.

Asynchronous RL solves the long-tail problem by allowing the inference policy to be updated as soon as the optimizer step is finished on the trainer deployment. In practice, this means the model being trained and the model generating rollouts can be slightly out of sync , which introduces its own set of problems that the team had to solve carefully.

Architecture diagram showing asynchronous RL with disaggregated trainer and inference pipeline

The inference stack: throughput over latency

Unlike production serving where you optimize for time-to-first-token, RL inference optimizes for raw throughput. The team stacked several techniques to get there:

  • FP8 inference with wide Expert Parallelism (EP). One of the best configurations to maximize throughput is Wide EP , large-scale expert parallelism, often spanning 32+ GPUs. To maximize throughput, they combine this with a large data-parallel rank, creating a large group of GPUs, each holding separate experts, each serving as a separate endpoint.
  • Prefill/Decode (P/D) disaggregation. Some combinations of model and environment produce as much as a 4:1 prefill-to-decode token ratio. Having the same inference workers serve both prefill and decode requests would increase end-to-end latency. With separate workers, long prefill requests , lengthy tool outputs, etc. , do not throttle the decode workers, letting them progress with predictable latency.
  • Centralized KV cache with Mooncake. Mooncake Store works as a centralized store that pools RAM and disk from all client nodes into one large pool, reachable from any inference worker on any node. This enables cross-replica prefix cache hits , a big deal when many rollouts share long common prefixes.
  • Smart routing. The NVIDIA Dynamo router scores each worker based on KV cache reuse, queue depth, KV cache utilization, and current load, then routes requests accordingly. Combined with Mooncake, you get load-balanced routing with cross-replica cache hits.

The hidden killer: trainer-inference KL mismatch

One of the most insidious problems in async RL is that the model generating rollouts and the model being trained slowly diverge. If the inference engine uses FP8 but the trainer uses BF16, they'll compute slightly different logits for the same input , and that gap accumulates over a long run, destabilizing training.

Prime Intellect attacks this on two fronts. First, FP8 training using DeepGEMM kernels to perform block-scaled FP8, as proposed by DeepSeek V3. This doesn't really increase throughput due to quantization overhead, but it heavily decreases KL mismatch between trainer and inference, as both now use the same precision and even the same kernels.

Second, they built Router Replay (R3): R3 works by capturing the routing decisions made during inference , which experts each token was sent to , and then replaying them directly on the trainer. This effectively reduces KL mismatch between trainer and inference by an order of magnitude, resulting in more stable training. The engineering challenge here is non-trivial: routed expert data is a large payload of shape [num_layers, top_k, seq_len] that can grow to hundreds of GB, requiring heavily optimized PyTorch operations to avoid CPU bottlenecks.

The training stack: 3D parallelism at 131k context

The trainer is built on TorchTitan and uses three parallelism strategies simultaneously , each solving a different memory or compute problem:

  • FSDP2 (Fully Sharded Data Parallel). Shards parameters, gradients, and optimizer states across ranks. Required at 1T+ scale just to fit the optimizer state in memory.
  • Expert Parallelism (EP). Instead of all-gathering the full MoE layer , which would require roughly 80GB of memory just for active layer weights , EP keeps experts sharded across GPUs and dispatches tokens using all2all primitives, significantly reducing active memory.
  • Context Parallelism (CP). At 131k+ sequence length, intermediate activations , not parameters , become the dominant memory cost. CP shards the sequence dimension across ranks. GLM-5's non-standard DSA attention required a custom CP implementation: K and V are gathered into a latent space (cheap), the indexer computes sparse indices for the full sequence, and core attention runs on those indices , only one all-gather collective per attention layer.
Architecture diagram showing expert parallelism with all2all dispatch and memory comparison: FSDP all-gather ~80GB versus EP ~10GB per layer

What's now possible

The practical upshot is that RL post-training of models that previously required enormous proprietary infrastructure is now accessible to anyone with a reasonably sized GPU cluster. prime-rl is built to train 1T+ MoE models on 1000+ GPUs with FSDP2 for training and vLLM for inference, with native integration with verifiers environments, including built-in support for SWE and agentic environments.

The optimizations also generalize beyond GLM-5. The optimizations translate to any large mixture-of-experts model, such as Kimi-K2.7-Code or NVIDIA Nemotron-3-Ultra-550B. Getting a training run started is a single command on a Slurm cluster:

code
uv run rl @ examples/glm5_llmd/rl.toml --output-dir /shared/outputs/glm5-llmd

The broader context matters here. The race to post-train frontier open-weight MoE models , GLM-5, Kimi-K2, Nemotron , is heating up fast, and inference-side efficiency is the key bottleneck separating groups that can iterate quickly from those that can't. The model, training frameworks, and environments are open-sourced under fully-permissive licenses (MIT and Apache 2.0), which means the community gets the full stack, not just the model weights.

Prime Intellect is also actively collaborating with the vLLM, Dynamo, and llm-d teams on routing and inference improvements, and is exploring speculative decoding, NVFP4 training, fault tolerance, elastic scaling, and sub-1-second weight transfer for future versions. The full performance deep dive is worth reading for anyone building or planning to build large-scale RL infrastructure.

Trending
  • No trending articles

Comments

avatar

Next Reads