Iowa State's CoRAP Catches AI Reasoning Models Giving Right Answers for Wrong Reasons
A new framework called CoRAP wraps reasoning models in statistical guarantees, ensuring the final answer is actually backed by valid step-by-step logic.
Takeaways
- New CoRAP framework provides finite-sample guarantees that reasoning models return answers backed by valid reasoning traces.
- Uses three quality functions (sequence, set confidence, conditional answer) calibrated via Learn-Then-Test.
- Hierarchical Shapley attribution identifies both training examples and specific reasoning steps behind uncertainty coverage.
- On CLEVR-Math with LMM-R1, average set size drops from 5.00 (CP-Router) to 2.76 at α=0.4.
- Influence functions replace retraining, making the method faster than full fine-tuning or LoRA as data scales.
- Read the full paper on arXiv for proofs and full experimental details.
Large reasoning models like the o-series and DeepSeek-R1 spend inference-time compute exploring solution paths, reflecting on errors, and refining intermediate steps before producing an answer. That extra thinking usually helps, but it also creates a subtle trust problem: a model can arrive at the correct final answer via completely broken reasoning, or produce fluent logic that leads to the wrong conclusion. A new paper from researchers at Iowa State University, Quantifying and Understanding Uncertainty in Large Reasoning Models, tackles this with a framework that ties uncertainty guarantees directly to the reasoning-answer structure.
Where conformal prediction falls short
Conformal prediction (CP) is the standard tool for producing prediction sets with finite-sample coverage guarantees, and it is distribution-free and model-agnostic. Existing approaches either calibrate the final answer while ignoring the reasoning trace, or treat the generation as a whole without verifying whether the logic actually supports the conclusion. For an LRM, a prediction set can technically cover the true answer while the underlying chain-of-thought is nonsense. The authors argue this defeats the whole point of using a reasoning model.
A second gap sits alongside the first. Even when uncertainty is bounded, practitioners rarely know why the model is reliable, meaning which training examples and which specific reasoning steps produced that reliability. Standard attribution methods treat training samples as atomic units and don't zoom into individual steps within a trace.