JiRackUltra_14b Brings a 14B Model to Local CPU Without the Cloud
A 14B ternary Qwen2.5-based model refactored with BitNet-style weights runs on plain CPUs with 8-12 GB of RAM and native tool-call plus robotics tokens.
- JiRackUltra_14b is a 14B Qwen2.5-based model refactored with BitNet ternary weights for CPU inference.
- Ships as GGUF in Q2_K, Q3_K_M and Q4_K_M, running from 6-12 GB of RAM.
- Tokenizer adds special tags for routing, tool calls and robotics workflows.
- Works out of the box with llama.cpp, Ollama, LM Studio, vLLM and Docker.
- Weights are MIT; prebuilt Docker and Ollama distribution costs $12 per user per year.
- Aimed at local agents, on-prem RAG and edge deployments without GPUs.
JiRackUltra_14b, a 14-billion-parameter model built around a Qwen2.5-style architecture, entered Hugging Face’s trending charts after its download counter reached roughly 991,000. Hugging Face counts artifact downloads rather than unique users, so automated and repeated pulls can contribute to that figure.
The repository targets local inference with conventional tools: GGUF files for llama.cpp and Ollama, several compression levels, and a tokenizer extended with tags for routing, tool calls, media, and robotics. That combination could simplify private RAG systems and CPU-hosted agents, although the release lacks the benchmarks needed to establish its quality and speed.
Inside the release
According to the model card, JiRackUltra_14b uses 48 transformer layers, a hidden size of 5,120, 40 query heads, eight key-value heads, and a vocabulary of approximately 152,000 tokens. The weights are listed under the MIT license.
The eight key-value heads implement grouped-query attention, which lets several query heads share cached keys and values. This reduces KV-cache memory compared with standard multi-head attention, though cache use still rises with context length, batch size, and concurrent requests.
| Variant | File size | Author’s RAM estimate | Intended trade-off |
|---|---|---|---|
| BF16 | 28.1 GB | 28 to 32 GB | Highest-precision published checkpoint |
| Q4_K_M | 10.1 GB | 8 to 12 GB | Recommended balance of size and quality |
| Q3_K_M | 8.42 GB | 7 to 10 GB | Lower memory use with a larger quality trade-off |
| Q2_K | 6.81 GB | 6 to 9 GB | Maximum compression |
The published RAM ranges describe the model files more closely than a complete inference process. The operating system, runtime buffers, KV cache, prompt length, and concurrency all add memory pressure. A 10.1 GB Q4 file, for example, cannot remain fully resident on an 8 GB machine alongside the operating system; memory mapping may allow execution at the cost of paging and lower throughput.
Run it with familiar runtimes
Recent Ollama builds can fetch the Q4 checkpoint directly from Hugging Face:
ollama run hf.co/CMSManhattan/JiRackUltra_14b:Q4_K_MRecent llama.cpp builds with Hugging Face download support can expose the same model through an OpenAI-compatible HTTP server:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
