Skip to main content
Back

Goodfire Maps 10 Million Neurons in Qwen3.6 to Challenge SAE Interpretability

Goodfire mapped every neuron in Qwen3.6, arguing that individual units, not learned features, are the right basis for interpretability.

  • Goodfire released a searchable atlas of 10 million neuron poles in Qwen3.6-35B-A3B
  • Keeping only the top 320 of 4,096 routed neurons per layer preserves most benchmark performance
  • A fine-tune on 100M tokens restores full reasoning after 92.2% activation sparsification
  • 47% of neuron poles have one mode; 69% have one dominant mode doing 90%+ of damage
  • Agent-written descriptions recover 73% of perfect-reader ceiling when simulating neuron activations
  • Companion paper Neurons as an Interpretable Basis argues neurons beat learned features as units of analysis

Goodfire has released a public neuron atlas covering 10 million neuron poles in Qwen3.6-35B-A3B, alongside research arguing that native neurons are more interpretable than commonly assumed. Much recent interpretability work has relied on sparse autoencoders because individual neurons appear to mix unrelated concepts. Goodfire reports that native neurons become more legible when researchers isolate their strongest activations and trace their causal connections through the model.

A pole represents one sign of a neuron’s activation, so a scalar neuron can contribute both a positive and a negative pole. The model uses a mixture-of-experts architecture, or MoE, which routes each token through a subset of specialized feed-forward blocks. The atlas labels both activation directions and places semantically similar labels near one another.

Coverage and depth differ across the release. Every pole receives an automatically generated label based on highly activating examples, while roughly 3,000 curated neurons include fuller circuit diagrams, mode analyses, and agent-written descriptions in the Library view.

Only 320 routed neurons survive each layer

For every forward pass, the researchers ranked the 4,096 routed MLP neuron activations in each layer and set the weakest 92.2% to zero. Keeping the top 320 per token produced only a 1.2–2.2 percentage-point decline on log-likelihood benchmarks that do not require generated reasoning. Ablation here means suppressing an activation during inference and measuring the resulting change in model behavior.

The retained set changes with every token, making this a dynamic activation result rather than evidence that a fixed 92.2% of neurons can be deleted from the model. It shows that most routed activations contribute little on a given forward pass; it does not establish an equivalent reduction in parameter count or inference cost.

Generated-reasoning benchmarks degraded under the same sparsification. A brief fine-tune restored the reported performance across language, question answering, code, and science tasks. Goodfire describes that run as less than 0.001% of the model’s original pretraining and post-training effort.

Nearly all retained activations fell above the 97th percentile or below the 3rd percentile of the activation distribution. The paper calls this high-magnitude region the p97/3 band, which becomes the main focus of its interpretability analysis.

A few causal jobs per pole

Goodfire measures polysemanticity through computation rather than response patterns alone. The researchers examine which upstream neurons drive a pole, which downstream neurons read its output, and how interventions affect the network. They call each coherent computational function a mode.

The analysis assigns exactly one mode to 47% of poles. Among poles with multiple modes, roughly half still have a dominant mode responsible for at least 90% of the causal damage observed when that pole is ablated. The reported picture is therefore one of limited, often strongly imbalanced polysemanticity.

SwiGLU gating can also help downstream components distinguish unrelated modes carried by the same neuron. One example responds to both marine biology and oral surgery. Contextual signals activate different downstream reader sets through their gate projections, allowing the same neuron-output direction to participate in different circuits.

An AI agent writes and tests the labels

Goodfire generated the atlas labels with Neuron Scientist, an agent harness that proposes a description, tests it against activation data, and revises its hypothesis. The paper evaluates those descriptions on four increasingly demanding tasks:

  • Detection: Given one activating snippet and one distractor, choose the activating example. Opus 5.5 reaches 95% accuracy for the strongest activations, compared with a 50% chance baseline.
  • Classification: Identify the active mode when a multimode neuron fires. Text-only classification reaches 88% macro-F1 at high activation levels.
  • Simulation: Predict the neuron’s activation token by token using only its written description.
  • Input synthesis: Generate text that activates the neuron more strongly than any example in a 10-billion-token natural-text corpus. The method succeeds for 94% of tested neurons.

For activations above the 99.9th percentile, description-based simulation recovers 73% of the improvement produced by a perfect reader that sees the true activations, according to the paper’s metric. The evaluator received no task-specific training for neuron simulation.

Chart comparing damage restoration from simulated neuron activations across activation percentile bands
Goodfire reports stronger restoration as the analysis moves toward the most extreme activation percentiles.

Q3 activations keep reasoning moving

SwiGLU neurons compute an activation from the product of a gated projection and an up projection. When both projections are negative, their product can become positive. Goodfire calls these double-negative outputs Q3 activations.

Suppressing Q3 activations caused little damage on many evaluations, yet produced a specific reasoning failure: the model repeatedly checked its work and looped instead of committing to an answer. Restoring the original Q3 values recovered reasoning performance.

Reversed Q3 sequences and Q3 activations taken from unrelated prompts also restored performance. Because token order and prompt identity had little effect in these interventions, the authors propose that Q3 activations supply aggregate statistical properties needed for reasoning. The experiments identify a causal role, while the underlying mechanism remains unresolved.

The atlas works as a discovery index

The public interface supports search and browsing across neighborhoods of related labels. Broad atlas entries are generated from maximally activating examples, so they function as hypotheses to test. The curated Library entries provide deeper evidence through modes, circuits, and intervention results.

A practical investigation can follow four steps:

  1. Search labels to identify candidate poles associated with a behavior or concept.
  2. Inspect activating examples and, where available, the curated mode and circuit analysis.
  3. Test the proposed behavior on held-out prompts and varied contexts.
  4. Use ablation, activation patching, or steering to measure causal effects.

Examples reported in the paper include:

  • A neuron that activates when a date crosses into a new month or year, even when the earlier date appeared hundreds of tokens before.
  • A neuron that encodes former political boundaries, activating for “Vilna, Poland” while remaining inactive for “Krakow, Poland.”
  • A neuron that responds to “Wait” when the model interrupts its own reasoning and can be steered to adjust self-doubt.
  • A neuron for the dedicatory “to” following “raise our glasses” in wedding toasts.

A challenge to SAE-first analysis

Sparse autoencoders, or SAEs, learn a new set of sparse features by recombining a model’s native activations. Researchers can then analyze those learned features instead of the original neuron basis, at the cost of training and validating an additional model artifact on a large activation corpus.

Goodfire’s interpretability essay argues that strong native activations already provide a useful basis when analysis incorporates causal roles, upstream drivers, and downstream readers. If the result generalizes, circuit discovery, probing, and steering could operate directly on model neurons for many investigations.

The Neuron Scientist experiments also offer a route to automating part of that work. Goodfire reports that description quality improves with stronger base models and additional agent rounds, shifting some effort from manual inspection toward repeatable, compute-driven analysis.

The result has clear boundaries

  • One model family: The experiments cover a Qwen3.6 MoE model with SwiGLU activations. Dense transformers, other gating functions, and other training regimes may organize computation differently.
  • Dynamic sparsity: The 92.2% figure describes per-token activation ablation. It does not identify a fixed set of removable neurons.
  • Fine-tuning dependence: Aggressive sparsification initially damages generated reasoning. Reported recovery requires an additional fine-tune.
  • Uneven interpretive depth: All 10 million poles receive rough automated labels, while about 3,000 neurons receive detailed circuit and mode analysis.
  • Unexplained weak activations: The work concentrates on the p97/3 band and leaves lower-magnitude activations largely uninterpreted.
  • Labels require validation: Agent-written descriptions predict behavior at high activation levels, but they remain empirical models of neuron function rather than complete mechanistic accounts.

The release provides a large test bed for a specific claim: strong native neuron activations can support useful causal analysis in this model. Replication across architectures, tasks, and activation ranges will determine how broadly that claim applies.