RadixArk's Miles v0.1 Cuts Trillion-Parameter RL Sync Time by 86%
RadixArk's open-source Miles framework runs asynchronous RL on a 744B-parameter model across 64 GB300 GPUs with a 263-second median step.
- RadixArk released Miles v0.1, an open-source production RL post-training system built on slime and SGLang.
- Trained a 744B-parameter GLM-5.2 model on 64 GB300 GPUs with a 263-second median asynchronous step.
- Peer-to-peer weight transfer cuts a 1T-parameter update from 53 to 7 seconds versus NCCL broadcast on 32 nodes.
- Token-in-token-out session server preserves exact sampled tokens across multi-turn agentic rollouts.
- Rollout Routing Replay records and replays MoE expert assignments to eliminate training-inference routing drift.
- Supports LoRA RL, on-policy distillation, true-on-policy alignment, MXFP8/NVFP4 precision, and diffusion models.
Miles v0.1 connects rollout and training for frontier-scale RL
RadixArk has released Miles v0.1, an open-source stack for reinforcement-learning post-training of large language models. It connects SGLang rollouts to Megatron-LM or PyTorch FSDP2 training, with exact token capture, mixture-of-experts routing replay, low-precision consistency checks, asynchronous execution, and three weight-synchronization methods.
Frontier post-training runs combine trillion-parameter mixtures of experts, long tool-using trajectories, and separate software stacks for inference and training. Small differences in kernels, precision, tokenization, or expert routing can distort policy updates without producing an explicit error. Miles extends THUDM’s lightweight slime framework, which has been used to train models including GLM-4.6.
One policy, two numerical worlds
Reinforcement-learning policy updates compare the probability recorded during rollout with the probability recomputed during training. When the two engines assign different log-probabilities to the same sampled token, the resulting importance ratio reflects implementation differences alongside the intended policy change. Persistent mismatches can destabilize training.
Miles assigns each part of that workflow to a specialized component:
| Layer | Implementation | Role |
|---|---|---|
| Rollout | SGLang | Generates trajectories at high throughput and returns tokens, log-probabilities, and expert assignments. |
| Training | Megatron-LM | Scales full-parameter training across large GPU fleets. |
| Alternative training | PyTorch FSDP2 | Trains Hugging Face model implementations directly. |
| Session handling | Miles session server | Preserves tokenization across multi-turn tool use. |
| Weight updates | Broadcast, P2P, or disk-delta | Moves updated parameters from training ranks to rollout engines. |
The three-stage loop covers rollout, training, and weight publication. It can run synchronously in lockstep or asynchronously, allowing rollout engines to continue generating while the trainer consumes completed trajectory groups. Runtime metrics expose step timing, prefix-cache performance, and rollout-to-trainer log-probability divergence.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
