---
title: "CGG Beats Trained Reward Models on Hard Math Without Any Training"
subtitle: "A new paper shows a large LLM can steer a small one during generation using only token probabilities, no reward model training required, matching PRM-guided search on math benchmarks."
url: "https://alphasignal.ai/news/cgg-beats-trained-reward-models-on-hard-math-without-any-training"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-01T00:00:00.000Z"
updated: "2026-06-09T09:04:27.293Z"
type: "Paper"
topics:
  - "Reasoning"
  - "Post Training"
subtopics:
  - "Test Time Compute"
  - "Math Reasoning"
  - "Distillation"
upvotes: 4500
source_url: "https://arxiv.org/abs/2606.01682"
comments: 0
truncated: true
---

# CGG Beats Trained Reward Models on Hard Math Without Any Training

> A new paper shows a large LLM can steer a small one during generation using only token probabilities, no reward model training required, matching PRM-guided search on math benchmarks.

Getting a small language model to reason reliably through hard math problems is one of the central challenges in inference-time scaling. The standard playbook involves either running the model many times and voting on the most common answer, or training a dedicated Process Reward Model (PRM) to score each reasoning step and guide the search. Both approaches have real costs. A new [paper by Atoosa Malemir Chegini and Soheil Feizi](https://arxiv.org/abs/2606.01682) proposes a third path: use a large off-the-shelf LLM as a process scorer, with zero additional training, by exploiting something every model already produces: token probabilities.

## Why the existing options fall short

Process reward models are specialized models that assign feedback to each intermediate step in multi-step reasoning, unlike outcome reward models (ORMs) which only score the final answer. That step-level feedback is powerful, but it comes at a steep cost. Training a PRM requires labels annotated at every intermediate step, presenting significant challenges for both manual and automatic data collection. OpenAI's PRM800K dataset, for instance, collected over 1 million step-level labels, with the filtered training set containing about 800,000 step-level labels across 75,000 solutions.

The simpler alternative, majority voting, avoids all that overhead. Majority voting samples multiple responses to the same prompt and selects the answer that occurs most often. But it has a fundamental flaw: if the small model consistently takes a wrong turn early in its reasoning, no amount of voting will fix it. All the samples are already committed to the bad path before any selection happens.

PRMs have proven effective at enhancing mathematical reasoning by leveraging increased inference-time computation. The question the paper asks is: can you get that same mid-generation steering without ever training a reward model?

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning](https://arxiv.org/abs/2606.01682)
- [Free Process Rewards without Process Labels](https://arxiv.org/abs/2412.01981)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving](https://arxiv.org/html/2408.00724)
- [Process Reward Models (PRMs): Calibration and Scaling — Emergent Mind](https://www.emergentmind.com/topics/process-reward-models-prms)
- [Let's Verify Step by Step (PRM800K)](https://arxiv.org/pdf/2305.20050)