VeRL-Omni Brings Open-Source RL Training to Diffusion and Omni-Modal AI
The VeRL-Omni Team pre-releases an open RL post-training framework for diffusion and omni-modality models, built on verl and vLLM-Omni.

- Pre-release: The VeRL-Omni Team has open-sourced VeRL-Omni, a general RL post-training framework for diffusion and omni-modality models.
- Built on verl + vLLM-Omni: Combines verl's training infrastructure with vLLM-Omni's high-throughput multimodal rollout engine, including step-wise continuous batching and embedding caching.
- Async reward overlap: Running the reward model (e.g., a VLM judge) on a dedicated GPU asynchronously cuts per-step wall-clock time by ~14%, and delivers ~25% higher end-to-end throughput vs. the diffusers-based flow_grpo baseline.
- FlowGRPO demo: Qwen-Image trained with an OCR reward reaches 0.96 validation reward from 0.69 in just 120 steps, with stable convergence curves.
- Model coverage: Qwen-Image is fully released; BAGEL and Qwen3-Omni-Thinker are PR-ready; Wan2.2 (video) and HunyuanImage-3.0 are in progress.
- Hardware: Supports both NVIDIA GPUs and Ascend NPUs; Apache 2.0 license; community meetings every Tuesday at 11 AM GMT+8.
Reinforcement learning post-training has become the go-to technique for squeezing alignment and task performance out of large language models. But the moment you step outside the world of text tokens and into diffusion models, video generators, and omni-modal systems that handle audio, images, and video simultaneously, the existing tooling largely falls apart. That gap is exactly what VeRL-Omni is designed to close.
The VeRL-Omni Team has announced the pre-release of VeRL-Omni, a general RL post-training framework for multimodal generative models. It is built on top of verl and vLLM-Omni, and the vLLM project is cheering it on as a key ecosystem extension. The framework is open-source under Apache 2.0 and available now on GitHub.
Why text-only RL tooling breaks for multimodal
While the LLM RL stack has evolved rapidly over the past year, multimodal generative RL, covering diffusion and omni-modality models for image, video, and audio understanding and generation, faces critical unmet needs. The core problem is structural: rollouts in diffusion models are denoising trajectories in a continuous latent space rather than token sequences, and a single rollout may invoke multiple heterogeneous model components and multi-stage pipelines, such as a text encoder feeding into a DiT feeding into a VAE. You cannot just plug a standard GRPO trainer into that pipeline.
On top of that, reward functions for multimodal tasks are themselves multimodal models. An OCR reward for a text-to-image model, for example, requires running a vision-language model to read the generated image and score it. Scheduling that reward computation efficiently, without stalling the training loop, is a non-trivial systems problem.
What VeRL-Omni actually ships

The framework is organized around three layers that work together: a rollout engine, a reward engine, and a set of training backends. Here is what each one brings:
- Rollout via vLLM-Omni: VeRL-Omni is built on top of verl and vllm-omni. vLLM-Omni handles multimodal generation with step-wise continuous batching and embedding caching, keeping GPU utilization high during the sampling phase.
- Async reward engine: vLLM is used to serve VLM-as-judge and OCR reward models. Critically, reward computation is overlapped with ongoing rollout and training, so the GPU running the reward model does not block the training loop.
- Modular training backends: The framework provides various trainers including DiffusersFSDP, Megatron, and VeOmni, with built-in optimization for diffusion and omni-modal models, allowing easy integration of different parallelism strategies such as FSDP, USP, and TP.
- Hardware flexibility: VeRL-Omni supports both NVIDIA GPUs and Ascend NPUs, allowing flexible deployment across diverse hardware backends.
The async reward trick that cuts wall-clock time
One of the most immediately practical results in the release is the async reward setup. In the Qwen-Image OCR demo, the team runs a Qwen3-VL-8B-Instruct model as the reward scorer. In the FlowGRPO example, Qwen-Image is trained with the OCR reward task, where the reward model scores each generated image by reading the rendered text and comparing it against the dataset ground truth.
Moving the reward model to its own dedicated GPU and running it asynchronously alongside training cuts per-step wall-clock time by approximately 14%. Across the full benchmark, VeRL-Omni achieves roughly 25% higher end-to-end throughput than the diffusers-based flow_grpo implementation, driven by vLLM-Omni rollout, FSDP training, and overlapped reward computation.
FlowGRPO: RL for flow-matching models
The first algorithm shipped in the release is FlowGRPO. Flow matching is a training paradigm for generative models that learns to move samples from noise to data along a straight path, and it underlies models like Qwen-Image. FlowGRPO is an online policy method for flow-matching models. It employs multi-step SDE sampling with a diffusion policy model to enable effective RL exploration, and adopts model-based rewards to assess generation quality. Think of it as GRPO, adapted so that the "actions" are denoising steps rather than token choices.

The training loop has four stages:
- Rollout Generation: The diffusion policy model generates sample rollouts, collecting trajectories of log probabilities and generated images.
- Reward Model Scoring: The reward model scores each generated sample, allowing the computation of trajectory advantages (how much better or worse each sample was than the group average).
- Policy Optimization: The policy is updated using a FlowGRPO CLIP-style loss, pushing the model toward higher-reward outputs.
- Weight Synchronization: The latest policy weights are periodically synced to rollout workers, so generated samples always reflect the current policy.
The results from the OCR training run are striking. Starting from a validation reward around 0.69, the model reaches 0.96 in just 120 training steps, with the rollout reward climbing from roughly 0.15 to 0.9 over the same window.

What models and algorithms are supported
The framework targets three families of models:
- Diffusion generators for image and video: Qwen-Image (released), Wan2.2 (WIP)
- Unified understanding + generation models: BAGEL (PR ready), HunyuanImage-3.0 (planned)
- Omni-modality models handling text, image, audio, and video jointly: Qwen3-Omni-Thinker (PR ready)
On the algorithm side, the released version ships with FlowGRPO, MixGRPO, and GRPO-Guard for Qwen-Image. BAGEL and Qwen3-Omni-Thinker support is PR-ready, with DanceGRPO for video generation and GSPO for omni-modal models on the roadmap.
What is still missing
This is a pre-release, and the team is candid about what is still in progress. The async reward overlap currently applies only to the reward computation step. A fully asynchronous pipeline across actor, rollout, and reward is on the roadmap but not yet available. Video generation support via Wan2.2 is work-in-progress, and the Megatron and VeOmni trainer backends are still being hardened. The documentation covers installation and the FlowGRPO quickstart, but coverage for other algorithms is still catching up.
Getting started
The repo is available at github.com/verl-project/verl-omni under Apache 2.0. Installation instructions and a FlowGRPO quickstart are in the docs. The team holds a weekly community meeting every Tuesday at 11:00 AM GMT+8 for roadmap discussion. If you are working on RL post-training for any diffusion or omni-modal model, this is the most complete open framework for that problem right now.