--- title: "VeRL-Omni Brings Open-Source RL Training to Diffusion and Omni-Modal AI" subtitle: "The VeRL-Omni Team pre-releases an open RL post-training framework for diffusion and omni-modality models, built on verl and vLLM-Omni." url: "https://alphasignal.ai/news/verl-omni-brings-open-source-rl-training-to-diffusion-and-omni-modal-ai" author: "AlphaSignal Newsroom" author_url: "https://alphasignal.ai/editorial-team" company: "vLLM" company_url: "https://alphasignal.ai/company/vllm_project" published: "2026-05-20T01:39:52.000Z" updated: "2026-06-16T21:27:01.351Z" type: "News" topics: - "Post Training" - "Image" subtopics: - "Rl" - "Vision Language" upvotes: 147 read_time: "5 min" source_url: "https://vllm.ai/blog/2026-05-14-verl-omni" comments: 0 --- # VeRL-Omni Brings Open-Source RL Training to Diffusion and Omni-Modal AI > The VeRL-Omni Team pre-releases an open RL post-training framework for diffusion and omni-modality models, built on verl and vLLM-Omni. Reinforcement learning post-training has become the go-to technique for squeezing alignment and task performance out of large language models. But the moment you step outside the world of text tokens and into diffusion models, video generators, and omni-modal systems that handle audio, images, and video simultaneously, the existing tooling largely falls apart. That gap is exactly what [VeRL-Omni](https://github.com/verl-project/verl-omni) is designed to close. The VeRL-Omni Team has announced the pre-release of VeRL-Omni, a general RL post-training framework for multimodal generative models. It is built on top of [verl](https://github.com/verl-project/verl) and [vLLM-Omni](https://github.com/vllm-project/vllm-omni), and the vLLM project is cheering it on as a key ecosystem extension. The framework is open-source under Apache 2.0 and available now on GitHub. ## Why text-only RL tooling breaks for multimodal While the LLM RL stack has evolved rapidly over the past year, multimodal generative RL, covering diffusion and omni-modality models for image, video, and audio understanding and generation, faces critical unmet needs. The core problem is structural: rollouts in diffusion models are denoising trajectories in a continuous latent space rather than token sequences, and a single rollout may invoke multiple heterogeneous model components and multi-stage pipelines, such as a text encoder feeding into a DiT feeding into a VAE. You cannot just plug a standard GRPO trainer into that pipeline. On top of that, reward functions for multimodal tasks are themselves multimodal models. An OCR reward for a text-to-image model, for example, requires running a vision-language model to read the generated image and score it. Scheduling that reward computation efficiently, without stalling the training loop, is a non-trivial systems problem. ## What VeRL-Omni actually ships ![VeRL-Omni architecture diagram showing Diffusion/Omni RL Trainers, Training System, and Multi-modal Generation and Reward Layer](https://vllm.ai/blog-assets/figures/2026-05-14-verl-omni/verl-omni-arch.png) The framework is organized around three layers that work together: a rollout engine, a reward engine, and a set of training backends. Here is what each one brings: - **Rollout via vLLM-Omni:** VeRL-Omni is built on top of verl and vllm-omni. vLLM-Omni handles multimodal generation with step-wise continuous batching and embedding caching, keeping GPU utilization high during the sampling phase. - **Async reward engine:** vLLM is used to serve VLM-as-judge and OCR reward models. Critically, reward computation is overlapped with ongoing rollout and training, so the GPU running the reward model does not block the training loop. - **Modular training backends:** The framework provides various trainers including DiffusersFSDP, Megatron, and VeOmni, with built-in optimization for diffusion and omni-modal models, allowing easy integration of different parallelism strategies such as FSDP, USP, and TP. - **Hardware flexibility:** VeRL-Omni supports both NVIDIA GPUs and Ascend NPUs, allowing flexible deployment across diverse hardware backends. ## The async reward trick that cuts wall-clock time One of the most immediately practical results in the release is the async reward setup. In the Qwen-Image OCR demo, the team runs a `Qwen3-VL-8B-Instruct` model as the reward scorer. In the FlowGRPO example, Qwen-Image is trained with the OCR reward task, where the reward model scores each generated image by reading the rendered text and comparing it against the dataset ground truth. Moving the reward model to its own dedicated GPU and running it asynchronously alongside training cuts per-step wall-clock time by approximately 14%. Across the full benchmark, VeRL-Omni achieves roughly 25% higher end-to-end throughput than the diffusers-based `flow_grpo` implementation, driven by vLLM-Omni rollout, FSDP training, and overlapped reward computation. ## FlowGRPO: RL for flow-matching models The first algorithm shipped in the release is FlowGRPO. Flow matching is a training paradigm for generative models that learns to move samples from noise to data along a straight path, and it underlies models like Qwen-Image. FlowGRPO is an online policy method for flow-matching models. It employs multi-step SDE sampling with a diffusion policy model to enable effective RL exploration, and adopts model-based rewards to assess generation quality. Think of it as GRPO, adapted so that the "actions" are denoising steps rather than token choices. ![FlowGRPO algorithm diagram showing prompt input through policy diffusion model, N-step SDE sampling, and trajectory outputs to reward models](https://vllm.ai/blog-assets/figures/2026-05-14-verl-omni/flowgrpo-algo.png) The training loop has four stages: 1. **Rollout Generation:** The diffusion policy model generates sample rollouts, collecting trajectories of log probabilities and generated images. 2. **Reward Model Scoring:** The reward model scores each generated sample, allowing the computation of trajectory advantages (how much better or worse each sample was than the group average). 3. **Policy Optimization:** The policy is updated using a FlowGRPO CLIP-style loss, pushing the model toward higher-reward outputs. 4. **Weight Synchronization:** The latest policy weights are periodically synced to rollout workers, so generated samples always reflect the current policy. The results from the OCR training run are striking. Starting from a validation reward around 0.69, the model reaches 0.96 in just 120 training steps, with the rollout reward climbing from roughly 0.15 to 0.9 over the same window. ![Line graph showing validation reward rising from approximately 0.69 to 0.96 across 120 training steps](https://vllm.ai/blog-assets/figures/2026-05-14-verl-omni/validation-reward.png) ## What models and algorithms are supported The framework targets three families of models: - **Diffusion generators** for image and video: Qwen-Image (released), Wan2.2 (WIP) - **Unified understanding + generation models:** BAGEL (PR ready), HunyuanImage-3.0 (planned) - **Omni-modality models** handling text, image, audio, and video jointly: Qwen3-Omni-Thinker (PR ready) On the algorithm side, the released version ships with FlowGRPO, MixGRPO, and GRPO-Guard for Qwen-Image. BAGEL and Qwen3-Omni-Thinker support is PR-ready, with DanceGRPO for video generation and GSPO for omni-modal models on the roadmap. ## What is still missing This is a pre-release, and the team is candid about what is still in progress. The async reward overlap currently applies only to the reward computation step. A fully asynchronous pipeline across actor, rollout, and reward is on the roadmap but not yet available. Video generation support via Wan2.2 is work-in-progress, and the Megatron and VeOmni trainer backends are still being hardened. The [documentation](https://verl-omni.readthedocs.io/en/latest/index.html) covers installation and the FlowGRPO quickstart, but coverage for other algorithms is still catching up. ## Getting started The repo is available at [github.com/verl-project/verl-omni](https://github.com/verl-project/verl-omni) under Apache 2.0. Installation instructions and a FlowGRPO quickstart are in the docs. The team holds a weekly community meeting every Tuesday at 11:00 AM GMT+8 for roadmap discussion. If you are working on RL post-training for any diffusion or omni-modal model, this is the most complete open framework for that problem right now. ## Sources - [X post by @vllm_project](https://x.com/vllm_project/status/2056912763172364350) - [Announcing VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models | vLLM Blog](https://vllm.ai/blog/2026-05-14-verl-omni) - [GitHub - verl-project/verl-omni: RL training framework for diffusion and omni-modality models](https://github.com/verl-project/verl-omni) - [Flow-GRPO: Training Flow Matching Models via Online RL | OpenReview](https://openreview.net/forum?id=oCBKGw5HNf)