Project Maya Runs 321B GLM-5.3-Flash on a Single Workstation GPU
Project Maya runs the 321B-parameter GLM-5.3-Flash on a single or dual NVIDIA GPU rig by streaming experts from VRAM, RAM, and NVMe.
- Project Maya runs the 321B-parameter GLM-5.3-Flash MoE on consumer NVIDIA rigs by tiering experts across VRAM, RAM, and NVMe SSD.
- Ships Maya-S, a 96.5 GB GGUF quant retaining 97.9% of FP8 zero-shot accuracy on ARC, HellaSwag, WinoGrande, PIQA.
- Hits up to 40 tokens/s decode and 440 tokens/s prompt on two Tesla V100 32 GB cards with 30 GB RAM.
- Minimum spec: NVIDIA compute 7.0+, 32 GB RAM, ~100 GB free on fast NVMe, Linux x86-64 with CUDA 12.x.
- Serves OpenAI and Anthropic-compatible APIs locally, works with Claude Code via
ANTHROPIC_BASE_URL. - Built on Strata and llama.cpp, MIT licensed; no Windows or WSL2 support yet.
Project Maya runs GLM-5.3-Flash on workstation hardware
Project Maya is an MIT-licensed inference stack that runs the 321-billion-parameter GLM-5.3-Flash mixture-of-experts model on Linux workstations. Its minimum target is one compatible NVIDIA GPU, 32 GB of system RAM, and a fast NVMe SSD, bringing a model that typically requires hundreds of gigabytes of accelerator memory within reach of local development hardware.
The project combines a custom inference engine, an OpenAI-compatible API, an Anthropic-compatible API, a browser dashboard, and quantization tools behind one setup script. Maya builds on Strata, rewriting its inference engine for GLM-5.3-Flash while retaining Strata’s server and dashboard.
| Component | Project detail |
|---|---|
| Model | GLM-5.3-Flash, 321B total parameters and about 18B active per token |
| Maximum context | Up to 1 million tokens, subject to configured memory limits |
| Quantized model | 96.5 GB Maya-S GGUF |
| Minimum GPU | NVIDIA compute capability 7.0 or newer |
| Host platform | Linux with at least 32 GB RAM and fast NVMe storage |
| Concurrency | One active request |
The model spills across three storage tiers
GLM-5.3-Flash uses a mixture-of-experts architecture, which divides parts of the network into specialized parameter groups called experts. A router selects a subset for each token, so the model activates about 18B of its 321B parameters at a time. The complete set of weights still needs storage, and Maya distributes those weights across GPU memory, system RAM, and SSD.
- GPU tier: Holds the most frequently selected experts, filling the available VRAM.
- RAM tier: Stores the next most active experts and allows CPU threads to compute some expert operations in place.
- SSD tier: Keeps the remaining experts on NVMe and loads them through memory-mapped and direct-I/O paths when requested.
Maya records which experts a workload selects and uses that history to warm the GPU cache. Repeated workloads can therefore keep their commonly used experts in faster memory while less active weights remain on the SSD.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
