IsValorum Squeezes a 35B Open-Source Qwen3.8 Model Into 14.65 GB
A 35B Qwen mixture-of-experts release strips refusal vectors and squeezes the whole model plus 256K context into 24GB of VRAM.
- IsValorum released an abliterated GGUF of empero-ai's Qwen3.8 35B MoE with zero refusals.
- Surgical per-tensor quantization hits 3.40 bpw with only +0.02 perplexity delta from FP16 baseline.
- Full 256K native context fits in under 20 GiB VRAM on a single 24GB GPU.
- Independent RTX 5090 testing sustained 247-251 tok/s single-stream generation.
- System RAM streaming delivers 24-28 tok/s on laptops with only 4.2 GB VRAM available.
- Ships with dedicated Q8_0 MTP speculative decoding head and vision projector companions.
Community GGUF compresses a 35B MoE model to 14.65 GB
IsValorum has published a community GGUF release that combines selective quantization with abliteration, an activation edit intended to suppress refusal behavior. The conversion packages empero-ai’s base model at roughly 14.65 GB while preserving a native 262,144-token context window. The repository reports WikiText-2 perplexity close to the unquantized model and zero refusals in a 10-prompt probe.
| Property | Value |
|---|---|
| Total parameters | 35.2 billion |
| Active parameters | Approximately 2.6 billion to 3.2 billion per token |
| Architecture | 40 layers, 256 micro-experts, 8 experts selected per token |
| Attention layout | 10 full-attention layers and 30 linear-attention or DeltaNet layers |
| Native context | 262,144 tokens |
| Average quantization | Approximately 3.40 bits per weight |
| Featured file size | Approximately 14.65 GB on disk |
| Format | GGUF for llama.cpp-compatible runtimes |
| Repository license | Apache-2.0 |
A 35B model with a 3B compute path
The mixture-of-experts architecture stores all 35.2 billion parameters but activates only eight experts for each token. That sparse routing reduces arithmetic during inference, although the complete weight set still occupies memory or must stream from system RAM.
Thirty layers use linear attention or DeltaNet-style recurrent state, while ten anchor layers use full attention. Standard transformer layers retain key-value data for every token, causing the KV cache to grow with context length. The recurrent layers maintain fixed-size state, so only one quarter of this model’s layers incur the usual KV-cache growth.
Refusal removal has limited evidence
Abliteration identifies activation directions correlated with refusal responses and projects those directions out of the model’s residual stream. The release describes its implementation as orthogonal activation steering, with the goal of changing refusal behavior while preserving routing, syntax, knowledge, and reasoning performance.
The repository reports a KL divergence of 0.0076 between the edited and baseline output distributions. KL divergence measures how much one probability distribution shifts from another, and a lower value indicates less change on the evaluated data. Interpreting that number requires the calibration set, layer-selection procedure, prompt set, and generation settings.
The reported behavioral probe produced refusals on 0 of 10 sensitive prompts, compared with a baseline refusal rate near 94%. Ten prompts establish a narrow result rather than comprehensive removal across languages, templates, sampling settings, and prompt categories. The “uncensored” label refers to the activation edit; actual behavior can still vary by request and runtime configuration.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
