---
title: "Northeastern's CATT Benchmark Exposes Why AI Data Attribution Misses Its Sources"
subtitle: "A new benchmark called CATT shows why data attribution methods fail, and two simple fixes boost recall by 35 to 72 points."
url: "https://alphasignal.ai/news/northeastern-s-catt-benchmark-exposes-why-ai-data-attribution-misses-its-sources"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-10-29T17:59:53.000Z"
updated: "2026-10-31T12:37:16.493Z"
type: "Paper"
topics:
  - "Llms"
  - "Benchmarks"
subtopics:
  - "Hallucinations"
upvotes: 166
source_url: "https://arxiv.org/abs/2610.40655"
comments: 0
truncated: true
---

# Northeastern's CATT Benchmark Exposes Why AI Data Attribution Misses Its Sources

> A new benchmark called CATT shows why data attribution methods fail, and two simple fixes boost recall by 35 to 72 points.

Language-model data attribution ranks training records by their estimated influence on a model’s answer. Developers can use those rankings to investigate memorization, audit data provenance, and identify examples that may require removal or correction. Yet popular methods often miss known source documents even when benchmark designers know exactly which records contain the answer.

A [Bau Lab paper](https://arxiv.org/abs/2610.40655) from Northeastern University isolates two causes: the loss function used to calculate gradients and the method for combining token scores into a document score. The researchers also introduce the Controlled Attribution Benchmark, or CATT, which separates these design choices from confounding properties of real-world corpora.

## Known facts, known sources

Real-world attribution evaluations mix duplicate documents, lexical lookalikes, paraphrases, and uncertain provenance. When a method fails, those overlapping variables make the cause difficult to identify. CATT creates fictitious knowledge and changes one dataset property at a time, giving each target fact a known training source.

The researchers fine-tuned Qwen3.5-4B on two synthetic tasks, then added controlled redundancy and distractors to test retrieval under progressively harder conditions.

__CATT tasks and controlled stressors__
| Component         | Construction                                                       | Purpose                                                                                      |
| ----------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- |
| Function Bindings | New functions always return an assigned value.                     | Tests whether attribution retrieves the records that taught a specific input-output mapping. |
| Routing Rules     | Invented rules sort objects according to their properties.         | Tests attribution on facts that require applying a learned rule.                             |
| Redundancy        | The same fact appears across N documents written in varied styles. | Measures whether influence becomes fragmented across repeated evidence.                      |

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Elucidating the Design Space of LM Data Attribution](https://arxiv.org/abs/2610.40655)
- [CATT Project Page](https://catt.baulab.info/)
- [CATT HTML paper](https://arxiv.org/html/2610.40655v1)
