Skip to main content
Back

Alibaba Maps 1,758 Safety Neurons Inside Qwen2.5 to Expose Hidden Jailbreak Paths

Alibaba's Safe-SAIL framework decodes safety-related neurons inside LLMs, releasing 1,758 interpretable features across four risk domains with a 55% cheaper explanation pipeline.

  • Alibaba's Safe-SAIL releases 1,758 interpreted safety neurons across pornography, politics, violence, and terror domains.
  • Segment-level simulation cuts explanation cost by 55% while retaining r=0.8 correlation with token-level ground truth.
  • New L₀,t and I_CDF metrics predict which SAE configuration produces the most interpretable safety features before expensive labeling.
  • Best configuration is TopKReLU with k=200 on Qwen2.5-3B-Instruct MLP layer 17; DeepSeek-R1 beats Claude 3.7 and QwQ-32B as the explainer.
  • Steering individual features drives unsafe response rate from 22% to 97.8%, evidence of causal (not just correlational) behavior.
  • Interactive ModelScope demo lets researchers query the feature database and visualize activations token by token.

Peering inside a language model to figure out which internal circuits recognize violence, or refuse to produce sexual content involving minors, has been an interpretability wishlist item for years. A team from Alibaba, Zhejiang University, Cambridge, and Renmin University just published a framework and open-source toolkit that takes a serious swing at it. Safe-SAIL trains sparse autoencoders (SAEs) on Qwen2.5-3B-Instruct, then explains and evaluates 1,758 individual safety-related neurons across pornography, politics, violence, and terror.

The interpretability gap for low-frequency concepts

Sparse autoencoders have become the go-to tool for mechanistic interpretability. They decompose entangled model activations into monosemantic features, meaning each latent ideally tracks a single human-understandable concept rather than a messy blend. The catch: under what circumstances SAEs derive fine-grained features for safety, a low-frequency concept domain, remains unexplored.

The authors call out two practical roadblocks. First, identifying which SAE configuration has the greatest potential for generating safety-specific features. Second, the prohibitively high cost of detailed feature explanation. Training a sweep of SAE configurations and then paying an LLM to write and verify explanations for every neuron in each configuration is out of reach for most labs. Prior releases like GemmaScope and LlamaScope trained the encoders but did not systematically produce large-scale, human-readable interpretations, especially for safety concepts.

Distribution of harmful content categories across the Safe-SAIL feature database

Picking the right SAE without breaking the bank

The first contribution is a pre-explanation evaluation metric that predicts how many usable safety features an SAE will yield before you spend money interpreting them. The team builds Concept Contrastive Query Pairs by prompting an LLM to rewrite a risky query into a matched safe version that preserves phrasing and structure. A feature that fires on the risky version but not its clean twin becomes a candidate for that concept.