Skip to main content
Back

Jackrong's Qwopus-GLM-18B Beats a 35B Model Running on Half the VRAM

A community frankenmerge stacks two 9B Qwen finetunes into an 18B model that outperforms a 35B MoE while fitting in 12 GB VRAM.

  • Qwopus-GLM-18B is a 64-layer frankenmerge of two Qwen3.5-9B reasoning distills, healed with a 1000-step QLoRA pass.
  • Scores 40/44 (90.9%) on a capability suite, beating Qwen 3.6-35B-A3B MoE (86.4%) at less than half the VRAM.
  • Passes 62/63 checks on frontend code stress tests, with balanced braces and parens across 6 apps.
  • Q4_K_M is 9.2 GB and fits in 12 GB VRAM, targeting RTX 3060 and 4070 class GPUs.
  • Runs out of the box via llama.cpp, Ollama, LM Studio, and vLLM under Apache 2.0.
  • QLoRA heal recipe: rank 64, 4-bit NF4, 1000 steps, ~14 hours on one RTX 5090, loss down 39%.

Someone glued two 9B language models on top of each other, ran a short healing pass, and ended up with something that beats a much larger mixture-of-experts model on a 44-test capability suite. That is the short version of Qwopus-GLM-18B-Merged-GGUF, an experimental open-weights model published under Apache 2.0 on Hugging Face.

The technique is called a frankenmerge, or passthrough merge: take all the transformer layers of model A, stack all the layers of model B directly on top, and end up with a deeper network that shares no additional training. This build is a 64-layer frankenmerge of two Qwen3.5-9B finetunes, stacking all 32 layers from each into an approximately 18B parameter model, then healed with a 1000-step LoRA fine-tune to smooth the layer boundary. The goal was pragmatic. Users have been asking for something between Jackrong's 27B and 9B models that runs well on 12 to 16 GB GPUs, and this frankenmerge fills that gap.

Stitching the seam

Stacking two independently trained models is cheap but ugly. Activations flowing out of layer 32 of model A were never designed to flow into layer 33 of model B, so the boundary tends to leak. The raw frankenmerge had a known symptom: garbled code output. Because two separately trained models were joined at layer 32, structured output like code blocks, HTML, and bracket matching would occasionally come out malformed or hallucinated.

The fix was a short, targeted QLoRA heal, a low-rank adapter trained in 4-bit precision just long enough to let gradients cross the boundary. The recipe:

  • QLoRA in 4-bit NF4, LoRA rank 64, adapters on every attention and MLP projection
  • Blended data: 70% reasoning traces, 15% competitive programming Python, 15% multi-step chat