TU Darmstadt Exposes How AI Circuit Research Has Been Measuring the Wrong Thing
A new paper shows that structurally different circuits in transformers can implement the exact same computation, exposing a fundamental flaw in how mechanistic interpretability evaluates circuit findings.
Takeaways
- Phantom specialization: Circuits that look structurally different across input conditions can implement the exact same computation -- a pattern the authors call phantom specialization.
- 75 circuits, one mechanism: Across 5 Pythia models (70M-1.4B) and 4 token-frequency bands, 75 extracted circuits share a universal core that recovers at least 99% of performance.
- Evaluation inflation: Source-level faithfulness evaluation inflates circuit accuracy by up to 0.85 points vs. edge-level evaluation, hiding the many-to-one mapping from structure to function.
- Circuits are not unique: Repeated extractions on the same data return structurally different circuits, suggesting algorithms sample from an equivalence class of valid subgraphs, not a unique mechanism.
- Fix requires two changes: Edge-level evaluation and cross-condition transfer tests are needed to distinguish real specialization from phantom specialization.
- No code released, but the setup uses standard open-source tools (ACDC + Pythia); full paper at arxiv.org/abs/2606.06267.
Mechanistic interpretability has a measurement problem. The field's core workflow -- find a circuit, compare it to another circuit, conclude they represent different mechanisms -- may be producing a systematic illusion. A new paper from the UKP Lab at TU Darmstadt introduces a term for it: phantom specialization. The circuits look different. The mechanisms are the same.
What circuits are, and why they matter
Circuit discovery identifies a sparse, directed cross-layer causal subgraph whose edges jointly mediate a target behavior and remain approximately faithful under interventions. Think of it as reverse-engineering a transformer by finding the minimal wiring diagram responsible for a specific behavior -- like completing a sentence, doing arithmetic, or copying a sequence. Such research is motivated by the circuit hypothesis, which posits that LLMs implement their capabilities via small subnetworks within the model.
The practical stakes are high. Mechanistic interpretability aims to reverse-engineer neural networks into human-understandable components and functional modules, offering fine-grained interpretability that serves purposes including transparency, trustworthiness, and safety. If circuits are the atomic unit of explanation, then comparing circuits across conditions -- different input types, different model sizes, different tasks -- is how researchers build understanding. That comparison step is exactly what this paper interrogates.
The hidden assumption nobody tested
Circuit discovery methods identify subgraphs that explain specific model behaviors, and structural differences between discovered circuits are commonly interpreted as evidence of distinct mechanisms. The authors test this assumption by varying input statistics while holding the task fixed, and show that the resulting structural differences exhibit apparent specialization but do not correspond to functional differences.
The task they chose is Literal Sequence Copying -- a model must reproduce a sequence of tokens verbatim. It is a clean, well-defined behavior with no ambiguity about what counts as success. The twist: they run the same task on four different token-frequency bands. High-frequency tokens (common words like "the") versus low-frequency tokens (rare words or subwords) produce inputs that look statistically very different, even though the underlying cognitive operation is identical.
The prediction from standard circuit interpretation would be: different input distributions produce different circuits, which means different mechanisms. The paper shows this prediction is wrong.