Peking University's Self-Critique Catches Benchmark Cheating That Fools Every AI Detector
A new method exposes cheating in reinforcement learning by asking the model to argue with itself, catching contamination that likelihood-based detectors miss entirely.
ProTakeaways
- Peking University researchers introduce Self-Critique, the first detector targeting RL-phase data contamination in LLMs.
- Existing likelihood-based detectors (PPL, Min-K%, Min-K%++) score near random guess on RL contamination because RL decouples training from likelihood signals.
- The method compares token-level entropy between an initial greedy response and a self-critique response asking for an alternative reasoning path.
- Contaminated samples cannot deviate due to RL-induced policy collapse, yielding highly similar entropy sequences under cosine similarity.
- Achieves up to 30% AUC improvement across AIME24, AIME25, Knights and Knaves, and SAT benchmarks, robust across PPO, GRPO, DAPO.
- Code and the new RL-MIA benchmark released at github.com/yongding-tao/RL-Data-Contamination.
Reinforcement learning has become the dominant recipe for pushing LLM reasoning performance, but a group of researchers from Peking University argue that it has also opened a blind spot in how we audit those models. Their new paper shows that virtually every existing data contamination detector fails to catch benchmark leakage introduced during the RL post-training phase, and proposes a technique called Self-Critique that closes the gap with AUC gains of up to 30 percent.
Audit tools built for the wrong training stage
Data contamination is the not-so-quiet scandal of LLM evaluation: when benchmark items sneak into training data, the reported numbers stop meaning what people think they mean. Detection methods exist for the pre-training and Supervised Fine-Tuning stages, but a critical gap opens up during Reinforcement Learning post-training, especially now that techniques like Reinforcement Learning with Verifiable Rewards (RLVR) are essential for improving LLM reasoning.
The reason existing detectors fail is structural. Pre-training and SFT are likelihood-based paradigms that train models to maximize the probability of observed data, naturally creating signals like unusually low perplexity that most current detectors are built to identify. RL, by contrast, operates on a reward-maximization principle, guided by sparse reward signals to find a successful reasoning path, which erases the very signals that traditional detectors rely on.
Put simply: perplexity-based tools like PPL, Min-K% and Min-K%++ look for evidence that the model has memorized a text sequence. RL doesn't teach memorization of sequences. It teaches the model to keep finding a rewarded answer, whatever path gets there. So the fingerprint you'd normally look for isn't in the log-probabilities anymore.
Policy collapse is the tell
The team's insight starts with a well-known side effect of RL training called policy collapse. Recent studies show that RL post-training frequently leads to policy collapse: for samples that receive consistent reward, the model converges to a narrow reasoning path, producing overly stable outputs, which shows up as sparse token-level entropy where many tokens are nearly deterministic.