RISC Beats Majority Voting Ceiling by 8% With a Tiny Ranker
RISC replaces majority voting with a learned ranker that cuts LLM calls by up to 82% while also beating peak self-consistency accuracy on QA benchmarks
- RISC replaces majority voting with a LightGBM LambdaRank model that scores answer candidates using five complementary features from the LLM's outputs.
- Up to 82% fewer LLM calls needed to match standard self-consistency accuracy on PopQA, with similar savings on HotpotQA.
- Beats self-consistency's ceiling by up to 7.98% on HotpotQA -- reaching accuracy levels majority voting structurally cannot achieve regardless of sample count.
- Five features capture answer frequency, semantic centrality (via MiniLM embeddings), reasoning-trace coherence, and shared intermediate reasoning steps across chains.
- Weaker on math tasks: gains on MATH500 are real but smaller, and cross-domain transfer to math from QA data degrades significantly.
- Open-source code available at github.com/s-nlp/RISC; works as a drop-in upgrade to any self-consistency pipeline without modifying the base LLM.
Self-consistency is one of the simplest and most effective tricks in the LLM playbook: sample the same question multiple times, then pick the most common answer. Instead of trusting a single chain, the method selects the answer most consistent across multiple reasoning paths, with the intuition that different valid reasoning chains should converge on the same correct answer. It works surprisingly well. The problem is that it wastes a lot of what it generates.
A large gap remains between majority voting and an oracle that always picks the correct answer from the sampled candidates. In other words, the right answer is usually already in the pile. The model just can't reliably find it. A new paper from researchers at AIRI, Skoltech, and EPFL proposes a fix: stop voting, start ranking.
The gap majority voting can't close
Several prior works have attempted to identify robust signals that help distinguish the correct answer among sampled candidates, including CISC, which relies on verbal uncertainty estimates, and methods based on internal geometric structure such as stable rank. These approaches all share the same fundamental design philosophy: find one good signal and use it. Such hand-crafted, knowledge-driven methods often struggle to scale and generalize to new tasks, a pattern long observed in AI research.
The new method, Ranking-Improved Self-Consistency (RISC), takes a different approach. Rather than picking a single signal and hoping it generalizes, it trains a small model to combine multiple signals together, treating answer selection as a ranking problem rather than a counting problem.
Five signals, one lightweight ranker
The core of RISC is a LightGBM LambdaRank model -- a gradient-boosted tree that learns to order candidates by correctness. LambdaRank is a classic learning-to-rank algorithm that optimizes directly for ranking quality (NDCG) rather than raw classification accuracy. The key insight is that it scores candidates relative to each other, not independently, which lets it exploit agreement structure across the full sample pool.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
