Skip to main content
Back

Tencent's DSRL Beats GRPO by Training AI Without Seeing the Question

Researchers propose reinforcement learning on the marginal distribution P(y) instead of P(y|x), then combine both stages to expand reasoning exploration beyond GRPO limits.

Paper, Full Access
  • New paper proposes PreRL, an RL method that optimizes the marginal distribution P(y) instead of P(y|x).
  • Theoretical and empirical results show 100% non-negative gradient alignment between log P(y) and log P(y|x).
  • Negative Sample Reinforcement in PreRL boosts reflection thoughts by 6.54x and transitions by 14.89x.
  • DSRL chains NSR-PreRL warmup with GRPO via a Policy Reincarnation strategy after 10 to 25 steps.
  • Beats GRPO, PPO, DAPO, Dr.GRPO on Qwen3-4B/8B across MATH500, AIME24/25, AMC23, OlympiadBench.
  • Out-of-domain gains: +5.37 on MMLU-Pro, +3.79 on GPQA-Diamond, +2.44 HumanEval at 8B.

A new paper from the Institute of Automation at the Chinese Academy of Sciences, the National University of Singapore, and Tencent AI Lab targets a well-known ceiling in reinforcement learning for reasoning models: the base model's own output distribution. The authors argue that if you only optimize the conditional distribution P(y|x), which is what standard RLVR pipelines like GRPO do, you're stuck refining trajectories the base model can already sample. Their proposal moves part of the optimization into what they call the pre-train space, updating the marginal P(y) directly with reward signals.

The framework has two components. PreRL (Pre-train Space RL) applies reward-driven online updates to the marginal distribution P(y) by masking the input question during gradient updates. DSRL (Dual Space RL) chains PreRL and standard RL together via a technique called Policy Reincarnation: warm up with PreRL to expand the reasoning horizon, then switch to GRPO for fine-grained refinement.

Diagram of Post-train, Pre-train, and Dual Space RL gradient objectives with training curves

Why touching P(y) at all is unusual

Prior work has shown that incentivized reasoning paths already exist within the base model's output distribution, which suggests the reasoning capacity of RLVR-trained models may be fundamentally bounded by the base model's capabilities. Standard RL sharpens the conditional policy over trajectories the base model already assigns non-trivial probability to. That's efficient but self-limiting.

Pre-training touches P(y), but through static corpora and next-token prediction, so no reward signal tells the model which reasoning patterns actually work. PreRL replaces passive data consumption with active, reward-guided learning on the marginal. The model generates rollouts, receives verifiable rewards, and updates P(y) online without conditioning on a specific question.