QUBRIC Breaks RL's Biggest Wall by Co-designing Questions and Grading Criteria

Reinforcement learning has become one of the most powerful tools for improving LLMs after initial training. But there is a hard wall: it works best when you can automatically check whether an answer is right or wrong. Math problems have a final number. Code has unit tests. Its success has thus far been largely confined to the mathematical and programming domains with clear and automatically checkable outcomes. For everything else, creative writing, legal analysis, moral reasoning, open-ended instruction following, the field has been stuck.

A new paper published on arXiv introduces QUBRIC, a framework that takes a fresh angle on this problem. Instead of trying to build better judges or smarter reward models, the researchers ask a more fundamental question: what if the training prompts themselves are the bottleneck?

The hidden flaw in rubric-based RL

The dominant approach for RL on open-ended tasks is rubric-based RL. The idea is to replace a binary verifier with a rubric: a set of criteria that a judge model can use to score responses. It sounds reasonable. But the QUBRIC authors identify a structural problem that prior work has missed.

Rubric quality is constrained by query structure. Open-ended queries yield vague rubrics; naively narrowing them introduces fabricated references that no model can verify, so all responses fail and training receives no reward signal. This is a training collapse: the model gets zero useful gradient and learns nothing.

The failure has two modes:

• Too open: A prompt like "write a persuasive essay" produces rubrics like "is the essay persuasive?" -- too vague to score reliably.
• Too narrow (naively): Rewriting it to "argue that X is better than Y using statistics from report Z" invents a report Z that does not exist, so every response fails the rubric check.