Adaptive RAG Flips Answers 55% of the Time When You Rephrase Questions
A new benchmark shows Adaptive RAG systems flip decisions and lose accuracy when the same question is worded differently by humans or models.
ProTakeaways
- New benchmark of 27K QA pairs tests Adaptive RAG on semantically identical query rewrites across six datasets.
- Retrieval decisions flip in up to 55% of cases under human paraphrases, even at temperature 0.
- Spelling errors cause the largest accuracy drops, up to 12.8 points on multi-hop 2WIKI.
- Larger models improve accuracy but not robustness; dense retrievers do not fix the gap.
- Human rewrites are shorter and higher-loss, triggering subquery drift and retrieval trajectory collapse.
- Both DRAGIN and Search-o1 paradigms are vulnerable; dataset and code to be released upon acceptance.
Ask a retrieval-augmented system the same question two different ways and you might get two different answers, two different token bills, and two entirely different retrieval traces. That is the uncomfortable finding from a new benchmark study on Adaptive RAG, the increasingly popular pattern where a language model decides on the fly whether it needs to look something up before answering.
The researchers built the first large-scale benchmark designed to probe how Adaptive RAG reacts to meaning-preserving rewrites, and the results are unflattering. Human rewrites cause decision flips in up to 55% of cases, revealing severe instability in retrieval decisions. Bigger models help with accuracy but do not close the robustness gap.
Why the adaptive gate wobbles
Standard RAG retrieves documents for every query, which wastes compute and often drags in noisy passages that hurt reasoning. Adaptive RAG tries to be smarter, dynamically deciding when and what to retrieve based on the query and evolving context, and skipping retrieval when the model can answer confidently from its own parameters. That decision is made by the model itself, and it turns out to be extremely sensitive to how the question is phrased.
The paper distinguishes two families of Adaptive RAG that dominate the literature. Logit-based methods watch model confidence and trigger retrieval when token probabilities or attention weights suggest uncertainty. Generation-based methods let the model decide during decoding by producing search queries as it writes. The authors test one representative from each: DRAGIN for the logit-based side and Search-o1 for the generation-based side.
Same question, different words
The core contribution is a benchmark of paraphrases that keep the semantic intent identical. The team assembled two kinds of variants: