Tel Aviv University's Faithfulness Serum Catches LLMs Lying About Their Own Reasoning
A Tel Aviv team shows LLM self-explanations often lie about the real reason for their answers, then patches the gap with attention-level attribution injection.
PRO- Tel Aviv team shows LLM textual explanations frequently fail to reflect the actual internal reasons for answers.
- Counterfactual hint-injection protocol proves unfaithfulness: models ignore metadata and grader-hack hints over 95% of the time.
- Faithfulness Serum injects LRP-based token relevance into attention scores at commitment layers (17-19 on Llama 3.1 8B).
- Training-free method roughly doubles faithfulness rates across MMLU, CommonsenseQA, SciQ, ARC-Challenge, OpenBookQA.
- Reveals authority bias: "Stanford professor" flips answers 2x more than "taxi driver" and gets 10x more credit.
- Accepted to ACL 2026 main conference; code released as supplementary material.
Ask a large language model why it answered the way it did and you will get a confident, well-formed rationale. New work from Tel Aviv University argues that rationale is frequently fiction: the model is post-hoc rationalizing an answer it actually chose for very different reasons. The paper, Faithfulness Serum, accepted to the ACL 2026 main conference, quantifies this gap and proposes a training-free fix that operates directly on attention scores.
Bar Alon, Itamar Zimerman, and Lior Wolf distinguish between two kinds of faithfulness. Subjective faithfulness means the rationale looks convincing to a human reader. Epistemic faithfulness, the property they actually care about, means the explanation reflects the internal evidence the model relied on for its decision. Their central finding is blunt: across open-source models like Llama 3.1 and Qwen 2.5, and closed models including GPT-4o and Gemini 2.0 Flash, explanations routinely fail this test.
Catching a model in the act
Proving an explanation is unfaithful is tricky because there is no ground truth for what the model was really thinking. The authors get around this with a counterfactual protocol borrowed from chain-of-thought monitoring work. They present a multiple-choice question, record the model's answer, then re-run the question with a hidden hint designed to flip the response. If it flips, the hint provably influenced the decision, so any honest rationale must mention it.
Four hint styles are tested:
- Sycophancy: A quoted authority figure claims the answer is X.
- Unethical information: Text framing the answer as coming from unauthorized system access.
- Metadata: The answer smuggled inside XML-like tags that look like machine-readable context.
- Grader hack: A snippet like
return answer == "A"that mimics a hidden validation function.
The numbers are damning. On Llama 3.1 8B with a general "what were the reasons behind your choice?" prompt, only 3.0% of explanations mentioned the metadata hint that had flipped the answer, and just 2.8% acknowledged the grader hack. Closed models did not save the day: on some hint types Gemini 2.0 Flash acknowledged the influence 0% of the time, and GPT-4o was similarly opaque on metadata manipulations.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
