Edge0 Squeezes a 145B Coding Agent Into a Single 24GB GPU
Edge0 ships a 145B latent Mixture-of-Experts model derived from Qwen3.8-Flash-Next that runs on a single 24GB consumer GPU at nearly 100 tokens per second.
- Edge0 released Edge0-145B-NVFP4, a 145B latent MoE running on one 24GB consumer GPU.
- Derived from Qwen3.8-Flash-Next, trained natively at 4 bits with 10.12B active parameters per token.
- Hits 98.3 tok/s agent decode and 86.5 Terminal-Bench 2.1 on RTX 5090 D.
- 512 experts per layer, 10 active, 256K context, SSD-streamed weights with predictive routing.
- Open-source inference stack under Apache-2.0; weights under restrictive Edge0 Community License 1.0.
- Beats base model on coding and terminal tasks, slightly behind on GPQA, HLE, IFBench.
Edge0 has released a 145-billion-parameter coding and tool-use model that runs on one NVIDIA GPU with 24GB of VRAM. The system fits by keeping selected experts on the GPU and moving others from host memory or SSD as needed. The reported configuration still consumes 31.28 to 38.27 GiB of system RAM, so the host memory and storage remain part of the serving footprint.
The release includes 145B and 120B models in NVFP4 and Q3_K formats, all derived from Alibaba’s Qwen base model. At publication, the 145B NVFP4 page showed more than 14,000 downloads. Download counts represent artifact pulls rather than unique installations or successful deployments.
| Model | Latent width | Formats |
|---|---|---|
| Edge0-145B | 1792 | NVFP4, Q3_K |
| Edge0-120B | 1280 | NVFP4, Q3_K |
Fast decode draws on RAM and SSD
Edge0 measured the 145B model on an RTX 5090 D with 24GB of VRAM. Prefill measures how quickly the engine processes an input prompt in parallel, while decode measures the sequential generation of output tokens.
| Metric | NVFP4 | Q3_K |
|---|---|---|
| Agent decode | 98.3 tok/s | 121.2 tok/s |
| 16K-token prefill | 3,165.7 tok/s | Not reported |
| GPU memory | 23.25 GiB | 23.14 GiB |
| System memory | 38.27 GiB | 31.28 GiB |
A cgroup-constrained test reduced the available resources to 16 GiB of VRAM and 20 GiB of system RAM. Edge0 reports 98.3 tok/s agent decode under that limit. The result shows how aggressively the engine can trade memory capacity for lower throughput.
A separate Edge0 chart lists 29GB of peak memory for the 145B model, compared with estimated figures of 70GB for a standard 4-bit Qwen3.8-Flash-Next build and 87GB for DeepSeek-V4-Flash. That 29GB number is a chart-specific peak metric. Deployment planning requires the separate GPU, system RAM, and storage figures reported above.
All performance numbers come from Edge0. Throughput will vary with batch size, prompt length, PCIe generation, SSD speed, expert-cache policy, and operating system. Independent reproductions would establish how the results transfer to other machines.
Latent experts shrink the working set
Edge0’s 145B architecture activates 10.12B parameters for each token. The active count limits arithmetic work, while the full 145B count determines storage and caching requirements.
- 48 transformer layers with a 2560-dimensional residual stream
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
