Toronto's STRIDE Traces AI Hallucinations to Training Data 12x Faster

Every time a language model hallucinates a math proof, generates a biased response, or confidently recites a memorized benchmark answer, the question is the same: which training examples caused this? That question is the domain of Training Data Attribution (TDA), and it has been stuck behind a wall of computational cost for years. A new paper from researchers at the University of Toronto, the Vector Institute, and the Max Planck Institute proposes a fundamentally different approach, one that sidesteps the gradient bottleneck entirely and achieves state-of-the-art results while being an order of magnitude faster.

The gradient wall

Training Data Attribution seeks to trace a model's predictions back to its training data. The cleanest way to do this is causal: retrain the model without a given example and measure what changes. But for LLMs, that is obviously impractical. The dominant paradigm is to approximate the counterfactual using parameter-space gradients, most notably via Influence Functions.

Gradient-based methods like LoGRA, TRAK, and TracIn all share the same fundamental problem. Applying gradient-based methods to LLMs introduces severe bottlenecks: materializing and storing gradients for billions of parameters across many examples requires enormous memory and compute. Worse, gradient methods rely heavily on the assumption that a model's loss landscape is locally convex and that the model has reached strict convergence; however, these assumptions are violated in deep learning. The alternative, representation-based methods that compare embeddings, are fast but lack causal grounding and particularly struggle to capture the complex dynamics of LLM pre-training.

Steering instead of differentiating

STRIDE (Steering-based Training Data Influence Decomposition) makes a conceptual leap: instead of asking "how would the weights change if we removed this example?", it asks "how would the model's internal activations need to shift to simulate that removal?" This is grounded in a key observation from the mechanistic interpretability literature: low-rank activation perturbations can faithfully replicate behavioral changes induced by fine-tuning.

STRIDE formulates training data attribution as a sparse recovery problem in the spirit of compressive sensing. It learns lightweight "steering operators" that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, it recovers individual training example influences via sparse linear decomposition.