CGG Beats Trained Reward Models on Hard Math Without Any Training
A new paper shows a large LLM can steer a small one during generation using only token probabilities, no reward model training required, matching PRM-guided search on math benchmarks.
- What it is: A training-free method called Chunk-Level Guided Generation (CGG) that uses a large LLM to steer a small LLM during generation using only token probabilities.
- No reward model needed: CGG matches or outperforms Qwen2.5-Math-PRM-72B guided search on most math benchmarks without any PRM training or step-level labeled data.
- Key result: CGS (the contrastive variant) beats majority voting by up to 28 percentage points; Qwen2.5-7B guided by Qwen2.5-72B hits 81.8% on MATH at k=16.
- Core mechanism: At each step, the small model samples k fixed-length candidate chunks; the large model scores them via log-probabilities (no text generation); the best chunk is committed.
- Length bias finding: Scoring variable-length reasoning steps with log-probabilities is systematically unreliable even after length normalization; fixed-length chunks fix this.
- Limitation: Requires two models loaded simultaneously; tested only on math reasoning; chunk length is a tunable hyperparameter.
Getting a small language model to reason reliably through hard math problems is one of the central challenges in inference-time scaling. The standard playbook involves either running the model many times and voting on the most common answer, or training a dedicated Process Reward Model (PRM) to score each reasoning step and guide the search. Both approaches have real costs. A new paper by Atoosa Malemir Chegini and Soheil Feizi proposes a third path: use a large off-the-shelf LLM as a process scorer, with zero additional training, by exploiting something every model already produces: token probabilities.
Why the existing options fall short
Process reward models are specialized models that assign feedback to each intermediate step in multi-step reasoning, unlike outcome reward models (ORMs) which only score the final answer. That step-level feedback is powerful, but it comes at a steep cost. Training a PRM requires labels annotated at every intermediate step, presenting significant challenges for both manual and automatic data collection. OpenAI's PRM800K dataset, for instance, collected over 1 million step-level labels, with the filtered training set containing about 800,000 step-level labels across 75,000 solutions.
The simpler alternative, majority voting, avoids all that overhead. Majority voting samples multiple responses to the same prompt and selects the answer that occurs most often. But it has a fundamental flaw: if the small model consistently takes a wrong turn early in its reasoning, no amount of voting will fix it. All the samples are already committed to the bad path before any selection happens.
PRMs have proven effective at enhancing mathematical reasoning by leveraging increased inference-time computation. The question the paper asks is: can you get that same mid-generation steering without ever training a reward model?
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
