RadixArk's Miles Tackles the Hardest Part of Building Large AI Models

Reinforcement learning has quietly become the hardest part of shipping a modern large language model. Once your base model is pretrained, the RL post-training loop has to juggle rollout servers, trainers, mixture-of-experts routing, weight synchronization, and low-precision arithmetic across hundreds of GPUs. RadixArk's new open source framework, Miles, is a bet that the answer is not another monolithic system but a thin PyTorch-native shell that stitches together the best pieces the community already has.

Miles composes SGLang for rollout, NVIDIA Megatron-LM for training, Ray orchestration, and PyTorch-native extensibility behind a small, pluggable trainer, with unified low-precision recipes, MoE-aware rollout/training alignment, fast NVIDIA NCCL/RDMA weight synchronization, observability, and fault tolerance built in. It is forked from and co-evolving with slime, the lightweight RL framework that has been battle-tested in large MoE training, where it is used to train GLM-4.6.

Why RL post-training became a systems problem

The framing in the Miles blog is worth pausing on. As models become larger, transition from dense to mixture-of-experts (MoE), and run across more distributed and specialized hardware, RL post-training is no longer just a training loop. It is a distributed systems problem. Rollout and training have almost opposite performance profiles: generation is memory-bandwidth bound because KV-cache reads dominate decoding, while training is compute and communication bound. Bridging them is where most in-house RL stacks break.

Miles addresses this with what its authors call a small-core, many-edges philosophy. The trainer itself is intentionally compact, and the parts researchers actually want to change plug in at launch time as ordinary Python modules.

How the four systems fit together

• SGLang runs the rollout engines that generate samples at high throughput.
• Megatron-LM is the training backend. Miles plugs directly into Megatron's argument parser, model-construction pipeline, training loop, parallelism primitives, and distributed checkpoint format rather than wrapping it as a black-box library.
• Ray handles orchestration. Every long-lived process, from trainer ranks to rollout servers to routing proxies, is a Ray actor, which gives Miles job submission, worker supervision, and dashboard visibility without extra infrastructure.
• PyTorch is the common numerical layer. Model components are regular torch.nn.Modules, losses are standard autograd graphs, and profiling stays inside familiar tools.

For bulk weight transfer after each training step, Ray handles the control path while the tensor bytes move over dedicated NCCL/RDMA channels. That keeps large tensor traffic off the Python data path, which matters a lot when you are shipping fresh weights to rollout workers every few seconds.

The MoE alignment trick

The most interesting technical piece is what Miles calls Rollout Routing Replay. In an MoE model, the expert router picks different experts for different tokens, and if the rollout policy and the training policy disagree about which experts fired, the trainer computes log-probabilities against a slightly different function than the one that produced the samples. That mismatch destabilizes RL on MoE models. Miles preserves the routing decisions across the rollout and training boundary so the trainer sees the same expert assignments the rollout used.

This pairs with a unified low-precision pipeline. Miles builds its low-precision pipeline on PyTorch's dtype system, with BF16, FP8, MXFP8, and INT4-QAT recipes that span training and rollout rather than living as isolated backend-only features. Because the sampling policy and the training policy have to stay numerically aligned, having one dtype story across both sides is more than a convenience.

What ships out of the box

The framework is not a research demo. According to the repo, it supports GRPO, GSPO, PPO and REINFORCE++ for RL, plus SFT and on-policy distillation, and it ships connectors for agentic environments including Harbor, HUD, NeMo Gym, OpenEnv, Verifiers and more, each plugging into the rollout layer that fits it, with task sandboxes on AgentENV, Daytona, E2B or Modal.

Model coverage tracks the frontier. Miles ships ready-to-run recipes for DeepSeek-V4, Kimi K2.5 and K2.6, GLM-5 and 5.1, and Qwen3.5 and 3.6, with support for NVIDIA Hopper and Blackwell GPUs. There is also a PyTorch FSDP2 backend available for runs that would rather train the HuggingFace implementation as-is, for teams that do not want to touch Megatron.

Extension points, not forks

The extension surface is where Miles diverges from the pattern of forking a framework every time a new algorithm or architecture appears. New architectures are handled through plug-in model specs, small files that insert custom PyTorch components, like a gated attention output module or a model-specific MoE router, directly into Megatron's model pipeline. That is how Miles claims to add support for DeepSeek-V3 and V4, GLM-4.7, and Qwen3 MoE variants without maintaining a long-lived Megatron fork.

On the RL side, the same idea applies. Users provide launch-time Python modules for:

1. Rollout functions for custom generation behavior.
2. Reward functions for task-specific supervision.
3. Loss functions for new RL objectives.
4. Sample filters for data selection and rejection.
5. Training hooks for auxiliary losses, metrics, and diagnostics.
6. Model specs for architecture-specific modules.

Async rollout and week-long jobs

Fully asynchronous mode is one of the more practical wins. In classic synchronous RL, the trainer stalls while rollouts generate and vice versa. In Miles, rollout actors stream samples continuously into a queue that the trainer drains at its own pace, eliminating the per-iteration blocking between the two phases. Combined with Ray-based rank-level fault tolerance, this is what makes week-long runs on hundreds of GPUs viable without babysitting.

There is also a speculative-decoding twist worth noting. The repo describes online SFT on the draft model during RL, instead of freezing it, avoiding draft policy drift away from the target model, and achieving 25%+ rollout speedup vs. frozen MTP, especially in later training stages.

Who should care

Miles is aimed at teams that are already running large-scale RL post-training and finding that the glue code between SGLang, Megatron-LM, and their own reward or agent infrastructure is where most of their engineering time goes. If you are running GRPO on a 200B-plus MoE, orchestrating agentic rollouts across sandboxes, or trying to keep FP8 numerics consistent between generation and training, this is the closest thing to a reference implementation the ecosystem has right now. If you are fine-tuning a 7B dense model with LoRA on a single node, TRL or Unsloth will still be simpler.

The bigger shift Miles represents is that RL post-training infrastructure is moving from siloed internal stacks at frontier labs into a composable, PyTorch-native open source layer. The pre-training war was won by open models. The post-training war may well be won by open orchestrators.