---
title: "HETA Beats Every Rival Method at Explaining Why LLMs Pick Specific Words"
subtitle: "A new attribution framework blends causal attention flow, second-order curvature, and information-theoretic masking to explain decoder-only LLM outputs."
url: "https://alphasignal.ai/news/heta-beats-every-rival-method-at-explaining-why-llms-pick-specific-words"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T23:09:13.945Z"
type: "Paper"
topics:
  - "Llms"
  - "Benchmarks"
subtopics:
  - "Hallucinations"
  - "Long Context"
upvotes: 1133
source_url: "https://arxiv.org/abs/2604.13258"
comments: 0
truncated: true
---

# HETA Beats Every Rival Method at Explaining Why LLMs Pick Specific Words

> A new attribution framework blends causal attention flow, second-order curvature, and information-theoretic masking to explain decoder-only LLM outputs.

Explaining why a large language model chose one token over another is still frustratingly hard. Most feature attribution methods were built for BERT-style encoders and rely on first-order gradients or attention weights, both of which quietly break down on autoregressive decoders. A new method called [Hessian-Enhanced Token Attribution (HETA)](https://arxiv.org/abs/2604.13258) proposes a different recipe: fuse causal attention flow, Hessian-based curvature, and KL divergence into a single token-level score, then evaluate it on a benchmark designed specifically for generative attribution.

## Why gradients and attention keep failing on decoders

The paper builds on a simple but underappreciated problem. Attention weights reflect where the model looks, not what actually affects the output, and can often be perturbed without meaningfully shifting predictions. First-order methods like Integrated Gradients depend heavily on baseline choice and can vanish entirely in flat regions of the loss surface, even when the function remains sensitive to finite perturbations.

The authors formalize this with a toy ReLU example: at an input sitting in the inactive region, both the pointwise gradient and Integrated Gradients along a baseline path return exactly zero, yet a small nudge across the hinge changes the output. In transformer LLMs, the same pathology shows up whenever saturating activations, residual routing, or causal masking hide influence from linear approximations. HETA is designed to close that gap.

## Three signals, one score

HETA computes a nonnegative attribution score for each context token `x_i` with respect to a target token `x_T` by combining three complementary signals:

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs](https://arxiv.org/abs/2604.13258)
- [HETA GitHub Repository](https://github.com/VishalPramanik/HETA.git)
- [HETA HTML paper](https://arxiv.org/html/2604.13258)
- [TellMeWhy: A Dataset for Answering Why-Questions in Narratives](https://arxiv.org/abs/2106.06132)