RePro rewards reasoning that moves an answer forward

Researchers at Shanghai AI Laboratory and Monash University have introduced RePro, a process-level reward for training reasoning models to produce shorter, more accurate solutions. The method rewards steps that increase the model’s confidence in the correct answer and penalizes loops, unproductive backtracking, and repeated derivations.

Long reasoning traces can improve performance on difficult tasks, but reinforcement learning often encourages models to spend thousands of tokens reconsidering conclusions they have already reached. RePro targets that training failure without imposing a blanket length penalty, allowing useful reasoning to remain while removing steps that make no measurable progress.

Outcome rewards miss the detours

Reinforcement learning with verifiable rewards, or RLVR, trains a model against answers that can be checked automatically. It is commonly used for mathematical reasoning and underpins systems such as DeepSeek-R1 and Kimi K1.5. A model generates a solution, receives a reward when its final answer matches the reference, and receives little guidance about the quality of individual steps.

That sparse signal permits inefficient traces to earn full credit. An earlier study found that o1-style models could generate thousands of reasoning tokens for elementary questions such as 2 plus 3. Typical traces contain repeated calculations, hedging, abandoned branches, and self-correction that leaves the answer unchanged.

The RePro authors describe these patterns through an optimization analogy: productive reasoning should steadily move the model toward the correct answer, while oscillation indicates stalled progress. This framing turns answer confidence into a measurable signal for each selected section of the trace.

Answer likelihood becomes a step signal

RePro measures progress using the probability assigned to the known answer after each partial reasoning trajectory. For a question