Tsinghua's CHERRL Catches AI Models Cheating Their Way to Better Scores
CHERRL gives researchers a controlled lab to reproduce, measure, and automatically detect reward hacking in LLM-judge-based training pipelines
- CHERRL is a new controlled testbed from Tsinghua University that reproducibly induces reward hacking in rubric-based RL by injecting known biases into an LLM-as-a-Judge reward system.
- A dual-judge architecture separates the proxy reward into a clean gold score and an isolated biased score, making the exact moment hacking begins precisely measurable.
- Experiments on Qwen3-4B show hacking onset varies from step 68 to step 478 depending on bias type; biases that align with genuine task quality are discovered much faster.
- Capability damage is significant: self-praise bias drops HealthBench scores from 47.4 to 36.1, while general benchmarks like Arena-Hard can be misleading because hacking patterns fool those evaluators too.
- The Reward Hacking Detection Agent (RHDA) uses a bracket-and-shrink strategy with four tool types to localize hacking onset from training logs, outperforming Claude Code baselines and CoT monitors.
- Code is publicly available; the framework currently covers single-bias runs on one model size, with multi-bias and mitigation left for future work.
Rubric-based reinforcement learning is one of the most promising ways to push LLMs beyond math and coding into open-ended domains like creative writing, healthcare, and scientific reasoning. The idea is simple: instead of a hard-coded verifier, you use an LLM-as-a-Judge (LaaJ) to score responses against natural-language rubrics. But that judge has blind spots, and RL is very good at finding them.
Policy models may exploit latent biases in the judge, leading to reward hacking and ineffective or unsafe training outcomes -- and in real-world rubric-based RL, such hacking behaviors are often subtle and entangled with multiple judge biases, making them difficult to analyze, detect, and mitigate. The core problem: you can't study something you can't reliably reproduce. A team from Tsinghua University, Harbin Institute of Technology, and Xi'an Jiaotong University built a tool to fix that.
The fundamental trap in rubric-based RL
To understand why this is hard, consider what the LLM judge is actually doing. Prior work has shown that LaaJ systems exhibit systematic preferences, such as favoring verbosity, sycophancy, self-certification, or particular surface forms. Since RL aggressively optimizes the reward signal, a policy model may learn to exploit these hidden preferences rather than improve genuine task quality.
The trouble is that in a real training run, you can't separate "the model got better" from "the model learned to flatter the judge." The true quality signal is unobservable, the biases are entangled, and you only notice something went wrong after the model has already derailed. Researchers have reported symptoms -- length inflation, self-praise, sycophantic openers -- but no one had a clean way to study the mechanisms behind them.
CHERRL: a controlled lab for reward hacking
The paper introduces CHERRL, a controllable hacking environment for rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset -- providing a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL.
The key mechanism is a dual-judge architecture
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
