---
title: "Fudan and Baidu's AAPO Fixes the Silent Flaw Stalling GRPO-Trained Reasoning Models"
subtitle: "A new reinforcement learning method fixes the zero-gradient problem plaguing GRPO by adding a reward margin against a frozen reference model."
url: "https://alphasignal.ai/news/fudan-and-baidu-s-aapo-fixes-the-silent-flaw-stalling-grpo-trained-reasoning"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T22:58:27.447Z"
type: "Paper"
topics:
  - "Reasoning"
  - "Post Training"
subtopics:
  - "Test Time Compute"
  - "Math Reasoning"
  - "Rl"
upvotes: 1164
source_url: "https://arxiv.org/abs/2505.14264"
comments: 0
truncated: true
---

# Fudan and Baidu's AAPO Fixes the Silent Flaw Stalling GRPO-Trained Reasoning Models

> A new reinforcement learning method fixes the zero-gradient problem plaguing GRPO by adding a reward margin against a frozen reference model.

Reinforcement learning has become the dominant recipe for teaching large language models to reason, but the workhorse algorithm behind DeepSeek-R1 and its imitators has a quiet efficiency problem. A team from Fudan University and Baidu's Frontier Research Department has published [AAPO (Advantage-Augmented Policy Optimization)](https://arxiv.org/abs/2505.14264), a drop-in replacement for GRPO that keeps training signals alive even when the policy gets stuck in the flat regions where GRPO effectively stops learning.

The paper reports consistent gains across Llama and Qwen model families on five math reasoning benchmarks, with the 7B variant hitting a 52.4 average on the standard suite. [Code is available on GitHub](https://github.com/JianxXiong/AAPO).

## The zero-advantage trap

[Group Relative Policy Optimization (GRPO)](https://arxiv.org/html/2505.14264), the algorithm DeepSeek popularized, replaces PPO's value network with a cheaper trick: sample a group of responses to the same prompt, score them, and use each response's deviation from the group mean as its advantage. It works, but it has a failure mode that gets worse as training progresses.

When a group's rewards are all similar, whether uniformly correct or uniformly wrong, the advantage approaches zero. Low variance within a group produces zero gradient and no parameter updates. The authors' statistical analysis shows this is common rather than an edge case: the zero-advantage proportion approached 100% in later training steps with increasing frequency.

The reason is intuitive. Once a policy learns the easy patterns, its samples for a given prompt cluster in quality. All responses look roughly the same to the reward model, group standard deviation collapses, and the normalized advantage goes to zero. The gradient goes with it. You are burning GPU-hours to compute updates of magnitude approximately zero.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin](https://arxiv.org/abs/2505.14264)
- [AAPO GitHub Repository](https://github.com/JianxXiong/AAPO)
- [GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning](https://arxiv.org/abs/2504.02546)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://arxiv.org/abs/2503.14476)
- [AAPO Full Paper HTML](https://arxiv.org/html/2505.14264)