Unsloth Shrinks Inkling-Small's 276B Model to Run on a Single Workstation
Unsloth shipped Dynamic 2.0 GGUF quants of Thinking Machines' 276B Inkling-Small, letting the multimodal MoE run locally in llama.cpp, Ollama and LM Studio.
PRO- Unsloth published Dynamic 2.0 GGUFs of Thinking Machines' 276B Inkling-Small model.
- Quants range from 1-bit UD-IQ1_S at 74.8 GB up to full BF16 at 527 GB.
- Sparse MoE means only 12B of 276B parameters activate per token, keeping throughput high.
- Runs in llama.cpp, Ollama, LM Studio, Jan, vLLM and Unsloth Studio with one-line commands.
- Base model scores 40 on AA Intelligence Index, 80.2% on SWE-Bench Verified, 89.5% GPQA.
- Multimodal input (text, image, 16 kHz audio) with 1M token context and Apache 2.0 license.
Unsloth has pushed out a full lineup of GGUF quantizations for Thinking Machines' recently released Inkling-Small, collapsing a 276-billion-parameter multimodal Mixture-of-Experts model down to sizes that fit on a single workstation. The Inkling-Small-GGUF repo has already crossed 885,000 downloads, which signals real appetite for running frontier open-weights locally rather than through an API.
Inkling-Small itself is an open-weights MoE with 276B total parameters and 12B active, roughly a quarter the size of its bigger sibling Inkling (975B total, 41B active). It was trained on NVIDIA GB300 NVL72 systems, reasons natively over text, images and audio, ships with a 1M-token context window and adjustable thinking effort, and releases under Apache 2.0.
Why the quants matter
Running Inkling-Small at native precision is not a hobbyist exercise. Per the model card, the BF16 checkpoint needs at least 600 GB of aggregated VRAM, met by 4x NVIDIA B300 or 8x NVIDIA H200. The NVFP4 checkpoint drops that floor to 180 GB, running W4A4 on a single B300 or W4A16 on two H200s.
Unsloth's GGUFs push the floor much lower using their Dynamic 2.0 quantization scheme, which selectively keeps sensitive layers at higher precision while aggressively compressing the rest. The available sizes:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
