MBZUAI's Adaptive Conformal Factuality Cuts LLM Hallucination Filter Errors Across Topics
A new adaptive conformal prediction method learns per-prompt thresholds via quantile regression, fixing over and under filtering of LLM claims across diverse topics.
PRO- New adaptive conformal prediction method rescales hallucination filter thresholds per prompt using embedding-conditioned quantile regression.
- Fixes over-coverage on easy categories and under-coverage on hard ones while preserving marginal guarantees.
- Tested on Mistral-7B, Llama-3.1-8B, and Gemma-3-12B across long-form QA and MMLU.
- Claim Conditioned Probability from LM-Polygraph chosen as best claim-level uncertainty scorer.
- Category coverage on Mistral tightened from a 73 to 85 percent spread to roughly 79 to 82 percent.
- Requires white-box model access and calibration data split three ways for quantile training, calibration, and testing.
Conformal prediction has become one of the more attractive tools for putting statistical guarantees on top of large language models, but it has a stubborn weakness: it treats every prompt the same. A team from MBZUAI is proposing a fix that lets the calibration threshold flex with the input, and the results suggest the standard approach has been quietly miscalibrating hallucination filters on heterogeneous data.
When one threshold fits nobody
The setup for conformal factuality is straightforward. You take an LLM response, break it into atomic claims, score each claim with some uncertainty measure, and drop the ones that exceed a threshold calibrated on held-out data. In theory, this gives you a statistical guarantee that a certain fraction of the retained claims will be factually correct.
The catch is what kind of guarantee you actually get. The authors point out that existing conformal procedures for LLMs lack adaptivity: a single calibrated quantile is applied uniformly across all test prompts, regardless of their difficulty, ambiguity, or rarity. That averages out fine across a whole dataset, but for any specific prompt category the threshold can be badly wrong. Easy topics get over-filtered, with too many good claims removed, while hard ones get under-filtered and let hallucinations slip through.
Rescaling by prompt difficulty
The proposed method, which they call Adaptive Conformal Factuality, keeps the split-conformal machinery intact but adds a learned rescaling step. The pipeline works like this:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
