GracefulRL trains language models to back off safely

Reinforcement learning with verifiable rewards can improve a language model’s accuracy while encouraging it to attempt every task. A shaky guess still offers some chance of reward, whereas abstaining offers none. That incentive becomes risky when an incorrect action can corrupt a database, execute a destructive command, or introduce faulty production code. Carnegie Mellon researchers propose GracefulRL, a training method that rewards models for using a safe fallback when success is unlikely.

Reinforcement learning with verifiable rewards, or RLVR, scores outputs using a checkable signal such as an exact answer, passing test, or valid game state. GracefulRL expands that signal into three ordered outcomes:

• Success: The model completes the task and receives a reward of 1.
• Fallback: The model executes a predefined safe action and receives a partial reward c, where 0 < c < 1.
• Error: The model attempts the task and fails, receiving a reward of 0.

A fallback might answer “I don’t know,” request human approval, stop a tool call, or move to a safe square in a game. The reward ordering makes completion preferable to abstention and abstention preferable to failure. The model learns which prompts warrant each behavior from outcome rewards, with no per-example labels specifying when to give up.

Why three rewards still collapse

Feeding the three rewards directly into Group Relative Policy Optimization (GRPO) or REINFORCE Leave-One-Out (RLOO) produces unstable behavior, according to the paper. GRPO compares rewards among multiple responses to the same prompt, while RLOO uses the other sampled responses as a baseline. Both depend on finite batches of model rollouts to estimate a useful policy gradient.