Community Strips Qwen-Image 2.1's Refusals to Run Locally on 16GB Laptops
A community-quantized, refusal-ablated text encoder for Qwen-Image-2.1 drops the refusal rate from 100% to 5% while running locally in ComfyUI.
- Refusal-ablated Qwen-Image-2.1 text encoder in GGUF, FP8 and bf16, 145K+ downloads.
- Refusal rate drops from 100/100 to 5/100 with KL divergence of only 0.0220 on benign prompts.
- Only the text encoder (Qwen3-VL-8B) is modified; DiT and VAE stay bit-identical to the official release.
- Q4_K_M variant runs in about 5 GB, making the full pipeline viable on 16 GB of unified memory.
- Requires the ComfyUI-GGUF-Qwen3VL-TE patch node to load the Qwen3-VL vision tower correctly.
- Companion GGUFs for the DiT and PE-T2I prompt rewriter complete an all-GGUF Qwen-Image-2.1 stack.
Qwen-Image 2.1 Gets a Refusal-Ablated GGUF Encoder
A community-built port of the Qwen-Image 2.1 text encoder has reached the top of Hugging Face’s trending list, with more than 145,000 downloads since release. The third-party model repository provides refusal-ablated weights in GGUF, FP8, and bf16 formats. When paired with compatible DiT and VAE components, these builds let developers run the image pipeline locally on consumer GPUs and Apple Silicon. Hugging Face’s counter includes repeat and automated file downloads, so it does not represent unique users.
Qwen-Image 2.1 uses Qwen/Qwen3-VL-8B-Instruct to convert prompts and reference-image context into conditioning for image generation. This port changes projection matrices inside that encoder. The diffusion transformer, which generates image latents, and the VAE, which converts those latents into pixels, retain their official weights. The encoder can replace the original component, although GGUF loading in ComfyUI requires the compatibility patch described below.
How the Refusal Direction Was Removed
The modification uses Heretic, a directional-ablation tool that estimates the activation direction associated with refusals and subtracts it from selected weights. The run targeted the encoder’s o_proj and down_proj matrices, which are Heretic’s defaults for this architecture.
The published evaluation is small and uses different sample sizes across checks. Its results cover the listed prompts and do not establish broad safety, benign capability, or generated-image quality.
| Check | Result | What it measures |
|---|---|---|
| Stock encoder on harmful prompts | 100 refusals out of 100 | Baseline refusal behavior |
| Ablated bf16 encoder on harmful prompts |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
