MBZUAI Study Exposes How 16 AI Confidence Methods Fail Under Real Uncertainty
A new benchmark reveals confidence scores in LLMs break down when questions have multiple valid answers or ambiguous phrasing, exposing predictable failure modes.
ProTakeaways
- New benchmark evaluates 16 UQ methods across 5 LLMs under three isolated uncertainty sources.
- Dataset contains 1,617 questions in triplets: single-answer, multi-answer, and ambiguous versions of the same base question.
- Existing methods drop 0.14 to 0.25 PRR points when uncertainty extends beyond model knowledge gaps.
- Output variability inflates uncertainty scores; input ambiguity deflates them, exposing symmetric, predictable failure modes.
- Models almost never request clarification on ambiguous inputs, instead committing to arbitrary interpretations with high apparent confidence.
- Experiments run through open-source lm-polygraph library covering token-probability, consistency, hybrid, and verbalized methods.
Uncertainty quantification (UQ) for large language models usually boils down to one number: how confident is the model that its answer is right? A new study from researchers at Mohamed bin Zayed University of Artificial Intelligence argues that this single-scalar view hides a serious problem. When uncertainty comes from something other than the model not knowing the answer, most popular UQ methods either overreact or go silent, and they do so in ways that are entirely predictable from how each method is built.
The team behind "Why Don't You Know?" built a controlled benchmark that isolates three distinct sources of uncertainty, then ran 16 UQ methods across 5 models through it. The result is a fairly damning picture of the state of confidence estimation in LLMs, and a useful diagnostic for anyone shipping models into production.
One score, three very different problems
The paper's central argument is that uncertainty in natural language generation can arise from fundamentally different sources: limited model knowledge, legitimate output variability, and ambiguous inputs. Each demands a different downstream action:
- Model knowledge uncertainty (MK): The question has one correct answer and the model may or may not know it. The right response is to abstain or consult an external source.
- Output variability (OV): The question has many equally valid answers ("name an animal that lives in the desert"). The model should answer confidently with any of them.
- Input ambiguity (IA): The question is underspecified. The model should ask for clarification.
Collapsing all three into one confidence number means a system cannot tell whether it should retrieve more knowledge, pick an answer freely, or ask a follow-up question. As the experiments show, the underlying scores are not just uninformative under OV and IA; they actively mislead.