---
title: "Alibaba's ProbeRAG Forces LLaMA to Stop Ignoring Retrieved Facts, Hitting 89.2 F1"
subtitle: "ProbeRAG cracks open the black box of RAG faithfulness by reading hidden states directly, boosting F1 by 3 to 10 points across benchmarks."
url: "https://alphasignal.ai/news/alibaba-s-proberag-forces-llama-to-stop-ignoring-retrieved-facts-hitting-89-2-f1"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-13T00:00:00.000Z"
updated: "2026-08-24T02:35:37.865Z"
type: "Paper"
topics:
  - "Retrieval"
  - "Post Training"
subtopics:
  - "Rag"
  - "Fine Tuning"
upvotes: 1733
source_url: "https://arxiv.org/abs/2510.12460"
comments: 0
truncated: true
---

# Alibaba's ProbeRAG Forces LLaMA to Stop Ignoring Retrieved Facts, Hitting 89.2 F1

> ProbeRAG cracks open the black box of RAG faithfulness by reading hidden states directly, boosting F1 by 3 to 10 points across benchmarks.

Retrieval-augmented generation was supposed to solve hallucinations. Instead, it produced a new failure mode: models that receive perfectly good context and still ignore it, either defaulting to what they memorized during pretraining or blending the two sources into something that resembles neither. A new paper from Xiamen University, Hong Kong Polytechnic, and Alibaba, titled [Beyond Black-Box Interventions](https://arxiv.org/abs/2510.12460), argues that fixing this requires looking inside the model rather than nudging it from the outside.

The team introduces [ProbeRAG](https://github.com/LinfengGao/ProbeRAG), a framework that reads a model's hidden states to detect when retrieved context conflicts with parametric memory, then steers attention toward the conflicting evidence during generation. On the ConFiQA multi-conflict benchmark, it pushes LLaMA-3.1-8B-Instruct from 28.0 F1 with vanilla RAG to 89.2 F1, with similar gains on Qwen3-8B and Mistral-7B-v0.3.

## Why RAG keeps ignoring its own evidence

The paper opens with a diagnostic that will feel familiar to anyone running a production RAG stack. Fed clean context, Qwen3-8B hits 71% accuracy. Add semantically related but topically irrelevant passages (contextual noise) and it drops to 59%. Introduce counterfactual entities that contradict what the model already knows (knowledge conflict) and it falls further to 56%.

Existing fixes fall into three camps, all of which treat the LLM as a sealed box:

* **Prompt-based methods** like Self-RAG or context-faithful prompting, which are brittle and prompt-sensitive
* **Decoding-based methods** like Context-Aware Decoding (CAD) or COIECD, which reweight token probabilities but break when context is noisy
* **Preference optimization** like Context-DPO or CANOE, which need large curated preference datasets

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generation](https://arxiv.org/abs/2510.12460)
- [ProbeRAG GitHub Repository](https://github.com/LinfengGao/ProbeRAG)
- [ProbeRAG full paper HTML](https://arxiv.org/html/2510.12460)