---
title: "QUBRIC Breaks RL's Biggest Wall by Co-designing Questions and Grading Criteria"
subtitle: "QUBRIC shows that rewriting training prompts alongside rubrics unlocks RL training for tasks where no ground-truth answer exists, gaining +5.5 points on ArenaHard"
url: "https://alphasignal.ai/news/qubric-breaks-rl-s-biggest-wall-by-co-designing-questions-and-grading-criteria"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-02T00:00:00.000Z"
updated: "2026-06-10T17:08:56.426Z"
type: "Paper"
topics:
  - "Post Training"
  - "Reasoning"
subtopics:
  - "Rlhf"
  - "Math Reasoning"
  - "Test Time Compute"
upvotes: 1920
source_url: "https://arxiv.org/abs/2606.03968"
comments: 0
truncated: true
---

# QUBRIC Breaks RL's Biggest Wall by Co-designing Questions and Grading Criteria

> QUBRIC shows that rewriting training prompts alongside rubrics unlocks RL training for tasks where no ground-truth answer exists, gaining +5.5 points on ArenaHard

Reinforcement learning has become one of the most powerful tools for improving LLMs after initial training. But there is a hard wall: it works best when you can automatically check whether an answer is right or wrong. Math problems have a final number. Code has unit tests. Its success has thus far been largely confined to the mathematical and programming domains with clear and automatically checkable outcomes. For everything else, creative writing, legal analysis, moral reasoning, open-ended instruction following, the field has been stuck.

A new paper published on arXiv introduces [QUBRIC](https://arxiv.org/abs/2606.03968), a framework that takes a fresh angle on this problem. Instead of trying to build better judges or smarter reward models, the researchers ask a more fundamental question: what if the training prompts themselves are the bottleneck?

## The hidden flaw in rubric-based RL

The dominant approach for RL on open-ended tasks is rubric-based RL. The idea is to replace a binary verifier with a rubric: a set of criteria that a judge model can use to score responses. It sounds reasonable. But the QUBRIC authors identify a structural problem that prior work has missed.

Rubric quality is constrained by query structure. Open-ended queries yield vague rubrics; naively narrowing them introduces fabricated references that no model can verify, so all responses fail and training receives no reward signal. This is a training collapse: the model gets zero useful gradient and learns nothing.

The failure has two modes:

* **Too open:** A prompt like "write a persuasive essay" produces rubrics like "is the essay persuasive?" -- too vague to score reliably.
* **Too narrow (naively):** Rewriting it to "argue that X is better than Y using statistics from report Z" invents a report Z that does not exist, so every response fails the rubric check.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards](https://arxiv.org/abs/2606.03968)
- [QUBRIC full paper HTML](https://arxiv.org/html/2606.03968v1)
- [Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation](https://arxiv.org/abs/2511.02463)
- [From Verifiable Dot to Reward Chain: RLVRR](https://arxiv.org/abs/2601.18533)
- [RLVR Beyond Math and Code: The Verifier Problem Nobody Has Solved](https://subhadipmitra.com/blog/2026/rlvr-beyond-math-code/)