vLLM Launches vime to Fix RL Fine-Tuning's Worst Numerical Drift Problem
vLLM's new open-source RL post-training framework bridges slime's battle-tested training stack with vLLM's inference engine for stable, hardware-flexible LLM fine-tuning.
- vime launched: vLLM's new open-source RL post-training framework, Apache 2.0, available now at github.com/vllm-project/vime.
- Built on slime: Inherits the battle-tested 3-stage architecture (Megatron training + data buffer + rollout) from slime, which powers GLM-4.5 through GLM-5.1.
- Key innovation: Replaces slime's SGLang rollout backend with vLLM, enabling stable train-inference alignment within the vLLM ecosystem.
- R3 routing replay: New technique for MoE models that replays expert routing decisions during training, cutting logprob drift by ~32% on Qwen3-30B-A3B.
- Hardware benchmarks: GB200 runs Qwen3-30B-A3B RL steps 1.72x faster than H200 (147s vs 252s per step) under the same framework.
- Ecosystem play: Positioned alongside NeMo RL, OpenRLHF, and verl -- not a replacement, but a vLLM-native option for teams already in that stack.
The vLLM team has launched vime, a new open-source reinforcement learning post-training framework that plugs directly into the vLLM ecosystem. The core idea is straightforward: take the training architecture from slime -- a framework already proven in production on frontier models -- and replace its inference backend with vLLM. The result is a single, unified pipeline for RL fine-tuning that inherits the best of both projects.
Why This Combination Matters
RL post-training (also called RLHF or RLVR) is the stage after pre-training where a model is fine-tuned using reward signals to improve reasoning, instruction-following, or tool use. It is computationally intensive because it alternates between two phases: rollout (the model generates responses, scored by a reward function) and training (the model's weights are updated based on those scores). Running both phases efficiently, on the same cluster, without the two sides drifting apart numerically, is the hard engineering problem.
slime is the RL framework behind GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, and GLM-4.5 -- a lineage of frontier models from Tsinghua's THUDM lab. It connects Megatron with SGLang and passes Megatron arguments through directly, so upstream training and serving optimizations remain available without adding another wrapper layer. The catch: slime uses SGLang as its rollout engine, not vLLM. For teams already invested in the vLLM ecosystem, that's a friction point.
vime connects slime's training stack with vLLM rollouts to provide a simple, stable, and efficient RL post-training pipeline. It is available now, open-sourced under Apache 2.0.
A Three-Stage Decoupled Architecture
vime inherits slime's three-stage design almost verbatim, with one key substitution at the rollout layer:
- Training (Megatron): Handles the main parameter update loop and syncs weights to the rollout side after each step.
- Rollout (vLLM + Router): Runs inference sampling to produce training samples, scored by reward or verifier signals.
- Data Buffer: Acts as the bridge between training and rollout, managing prompt injection and custom rollout logic.

This decoupled design means training and inference can run on separate GPU pools (non-colocated) or share the same hardware (colocated), depending on your cluster. At the framework level, training resources, rollout resources, and cluster topology are abstracted uniformly, making it easier to reuse the same RL pipeline across different hardware backends as support evolves with the vLLM ecosystem.
The Train-Inference Alignment Problem
One of the trickiest bugs in RL post-training is silent numerical drift between the training model and the rollout model. When the model generating samples (rollout) diverges from the model being trained, the gradient updates become stale and training destabilizes. This is measured by train_rollout_logprob_abs_diff -- the absolute difference in log-probabilities between the two sides.
For Qwen3-4B on A100 with GRPO, 4 training + 4 inference non-colocate, and gsm8k, vime's train_rollout_logprob_abs_diff stays stable around 0.011 throughout training. The baseline drifts continuously to around 0.77 as training progresses.
That's a 70x gap -- the kind of drift that quietly corrupts a training run.
For Mixture-of-Experts (MoE) models, the problem is even worse because different tokens get routed to different experts during inference versus training, compounding the mismatch. vime introduces a technique called R3 (routing replay) to address this: it replays the expert routing decisions from rollout during the training step, so both sides see the same token-to-expert assignments. Enabling R3 routing replay reduces the logprob diff from roughly 0.019 to roughly 0.013 for Qwen3-30B-A3B MoE, markedly reducing MoE train-inference mismatch.

Hardware Numbers
The team ran benchmarks on both NVIDIA H200 and GB200 hardware. For Qwen3-30B-A3B with 8-GPU colocate, dapo-math-17k, and GRPO, GB200 mean step time is about 147 seconds, while H200 mean step time is about 252 seconds -- GB200 end-to-end step speed is about 1.72x that of H200.

What's Supported Out of the Box
- Algorithms: GRPO (Group Relative Policy Optimization) and PPO, with CI-verified end-to-end examples.
- Models: Qwen3 Dense, Qwen3 MoE, and GLM-4.5-Air, with example scripts in the
scripts/andexamples/directories. - Deployment modes: Colocated (training and rollout share GPUs) and non-colocated (separate GPU pools).
- Hardware: A100, H200, and GB200, with more backends planned via vLLM's hardware plugin system.
The parameter system is intentionally familiar to anyone who has used slime or Megatron. vLLM-side arguments are passed through using the --vllm- prefix, keeping the control surface close to the upstream engines without adding abstraction layers.
Where It Fits in the Ecosystem
The vLLM team is explicit that vime is not meant to replace the other frameworks in its ecosystem. The vLLM community supports a broad set of LLM post-training frameworks, including NeMo RL, OpenRLHF, verl, and others. vime was built to seamlessly bring slime's proven training paradigm into the vLLM ecosystem, offering a production-ready bridge that aligns both projects' rapid release cycles.
The positioning is clear: if you are already using vLLM for inference and want to run RL post-training with a lightweight, stable framework that does not require learning a new abstraction layer, vime is the path of least resistance. Teams that need deeper async pipelines, multimodal RL, or tighter Ray integration may still prefer verl or OpenRLHF.
What's Coming Next
The roadmap covers three areas:
- Deeper vLLM integration: Router support, prefill-decode (PD) disaggregation, FP8 quantization, and multi-model serving.
- More hardware: Extending backends via vLLM's hardware plugin system.
- Training algorithms: Fully asynchronous pipelines, agentic RL for multi-turn tool calling, and support for new architectures like MoE and VLMs.
vime is free, Apache 2.0 licensed, and available now at github.com/vllm-project/vime. The quick start docs walk through configuring Megatron training resources and vLLM rollout resources, then launching train.py or train_async.py.