Tencent's UniRL Open-Sources One RL Loop for Every AI Model Type

Tencent's UniRL lets one RL training loop post-train diffusion models, LLMs, VLMs, and hybrid architectures — with two new algorithms included

·
·
·
Tencent's UniRL Open-Sources One RL Loop for Every AI Model Type
  • UniRL open-sourced: Tencent Hunyuan releases a unified RL post-training framework covering LLMs, VLMs, diffusion, video, and hybrid models in one loop.
  • Two new algorithms: Flow-DPPO (exact-divergence trust-region for flow/diffusion models) and DRPO (smooth quadratic regularizer for LLM RL) ship with the release.
  • Broad model support: SD3, FLUX, WAN, HunyuanVideo, Qwen3, Qwen-VL, HunyuanImage3, and Bagel all supported out of the box.
  • Unified model RL: First open-source framework to support RL on hybrid autoregressive+diffusion architectures like HunyuanImage3 and Bagel.
  • Production-ready infra: FSDP2 sharding, pluggable rollout engines (SGLang, vLLM-Omni), and three deployment modes from a single config.
  • Available now: Free and open-source under Apache 2.0 at github.com/Tencent-Hunyuan/UniRL.

Reinforcement learning post-training has become one of the most powerful tools in modern AI, but there's a catch: almost every RL framework is built for exactly one type of model. You need one stack for your LLM, another for your diffusion model, and yet another if you're working with a unified architecture that combines both. UniRL, just open-sourced by Tencent Hunyuan, is a direct attack on that fragmentation.

One loop to rule them all

UniRL applies one RL post-training loop , generate samples, score them, compute advantages, update the policy, and sync weights back to rollout workers , across multimodal model families. That sounds simple, but the engineering challenge is significant: diffusion models, autoregressive LLMs, and vision-language models all have fundamentally different generation mechanics, probability representations, and training dynamics.

The key design insight is that model family and algorithm are treated as two independent axes. Any supported algorithm can run on any supported model in its domain, so the total coverage is the product of the two dimensions rather than a fixed list of hand-matched pairs. The loop is always the same: generate, score, advantage, update, sync.

The framework is layered and composable. Each entrypoint (train_diffusion, train_ar, train_pe, train_unified_model) loads a Hydra config covering model, algorithm, rollout, reward, placement, and sync, then creates the matching domain trainer. The trainer coordinates the RL loop across pluggable rollout engines, algorithms, model bundles, reward services, and the shared distributed runtime: Ray DevicePool, FSDP2, Transfer Queue, and LoRA/full-weight sync.

The model zoo is unusually broad

The supported model table is where UniRL's scope becomes clear. It covers:

  • Image diffusion: Stable Diffusion 3/3.5, FLUX.2-Klein, Qwen-Image
  • Video diffusion: WAN 2.1/2.2, HunyuanVideo 1.0/1.5
  • Vision-language AR: Qwen-VL
  • Text-only LLM: Qwen3
  • Prompt-enhancer (LLM + diffusion): a rewriter that improves prompts before passing them to a diffusion model, trained end-to-end with a diffusion reward
  • Unified AR + diffusion: HunyuanImage3 and Bagel , a model class that no single-purpose RL repo can even express

That last category is the most novel. UniRL supports unified autoregressive + diffusion generation models like HunyuanImage3 and Bagel , a model class no single-purpose RL repo can even express. These models use an autoregressive component for reasoning and a diffusion component for generation, and until now there was no RL infrastructure that could handle both simultaneously in one training loop.

Two new algorithms ship with the release

UniRL also includes standard reference algorithms , GRPO, DiffusionNFT, DanceGRPO, and MixGRPO , but the headline additions are two team-original algorithms that address known weaknesses in existing approaches.

Flow-DPPO tackles a fundamental problem with applying RL to flow matching models (the architecture behind many modern image and video generators like FLUX and HunyuanVideo). Due to the deterministic nature of flow-matching models, they are not intrinsically designed for reinforcement learning. The probability flow ODE deterministically maps inputs to outputs, leaving little room for the stochastic exploration that reinforcement learning requires. This mismatch makes direct application of standard policy optimization methods ineffective.

Flow-DPPO solves this with a trust-region mask based on exact divergence computation rather than the noisy probability-ratio approximations used in standard PPO. The divergence mask blocks gradient updates that would push the policy too far from the reference , but only when that divergence can be measured precisely, giving the optimizer a principled safety boundary instead of a heuristic one.

DRPO (Divergence Regularized Policy Optimization) addresses a different problem: the instability of PPO's ratio-clipping mechanism when applied to LLMs with large, long-tailed vocabularies. The probability ratio used in PPO is highly volatile for low-probability tokens. In contrast, TV divergence is more stable. This highlights a key flaw of PPO's clipping mechanism: it over-penalizes low-probability tokens, which can slow down learning; and under-penalizes high-probability tokens, which can permit large, destabilizing updates.

DRPO replaces the clipping heuristic with a smooth, advantage-weighted quadratic regularizer at the token level. Rather than hard-clipping the policy ratio, it applies a penalty that scales smoothly with how much the update deviates from the reference , avoiding both the instability of large updates on common tokens and the over-suppression of rare ones.

Built to scale in production

The infrastructure side is designed for real training runs, not toy experiments. Three deployment modes are available from a single config knob, and rollout engines are pluggable behind a typed contract , you can swap between train-side rollout, SGLang, and vLLM-Omni without changing your training code. FSDP2 sharding handles large model parallelism.

Getting started on a single node looks like this:

bash
# Run a compose-check, then launch a single-node SD3 experiment
python -m unirl.train_diffusion --config-name=diffusion/sd3_trainside --cfg job --resolve
bash examples/run_experiment_single_node.sh diffusion/sd3_trainside
# Or launch an LLM experiment with DRPO on Qwen3
bash examples/run_experiment_single_node.sh ar/qwen3_drpo_4b_base_dpao_sglang

The repo acknowledges its foundations openly , UniRL builds on ideas and infrastructure from the open-source RL and inference ecosystem, especially vLLM, SGLang, slime, and verl.

Why this matters now

The broader context here is the rapid convergence of generative model architectures. Unified models that combine autoregressive reasoning with diffusion-based generation are becoming a serious research direction, and teams working on them have been forced to either build custom RL infrastructure or stitch together incompatible frameworks. UniRL is the first open-source framework to treat this as a first-class problem.

There are real limitations to note. The repo is freshly released with only 5 commits and no formal versioned releases yet. Algorithm coverage for newer model families like FLUX.2-Klein, HunyuanVideo 1.5, and Bagel is still being expanded. And while the architecture is clean, the complexity of configuring multi-node distributed runs across heterogeneous model types will require careful setup. The roadmap explicitly lists broadening algorithm-to-model coverage as a near-term priority.

For teams working on text-to-image alignment, video generation quality, or LLM reasoning , and especially for anyone experimenting with unified architectures , UniRL offers something that didn't exist before: a single codebase where the RL training loop is not the bottleneck. The DRPO paper and the Flow-DPPO manuscript are both available for the algorithmic details, and the official documentation covers the full launch guide.

Trending
  • No trending articles

Comments

avatar

Next Reads