Gradient's Parallax Turns Idle Macs and Gaming PCs Into a 235B AI Cluster
Gradient's open-source Parallax turns a pile of mismatched Macs and GPUs into a single inference cluster, hitting 3.1x throughput over Petals.
ProTakeaways
- Gradient released Parallax, an Apache-2.0 distributed inference engine for heterogeneous clusters.
- Shards LLMs via pipeline parallelism across NVIDIA GPUs and Apple Silicon using SGLang, vLLM, and MLX backends.
- Benchmarks show 3.1x throughput and 5.3x lower inter-token latency versus Petals on Qwen 72B/235B.
- Supports 40+ models including DeepSeek V3.2, Kimi-K2, Qwen3-Next, GLM-5, gpt-oss, and Llama 3.
- P2P layer built on Hivemind DHT; nodes join a Swarm via
parallax runandparallax join. - Read the technical paper for scheduler and paged KV cache internals.
Renting a datacenter GPU to serve your own LLM is expensive, and most of us have laptops, gaming rigs, and Mac Studios sitting mostly idle. Parallax, an open-source project from Gradient, tries to close that gap by letting you stitch those machines together into a single distributed inference cluster, regardless of whether they run CUDA or Apple Silicon.
It bills itself as a fully decentralized inference engine for building your own AI cluster across nodes with varying configurations and physical locations. The project is Apache 2.0, and the code, docs, and a technical paper are all public.
Shard the model, not your budget
Rather than loading a 70B model onto one big GPU, Parallax splits it across a mesh of peers and streams activations between them. The system runs on a distributed architecture called the Swarm, where your prompt is tokenized and executed across peers. The motivation is blunt: most researchers and developers cannot afford large GPU clusters, powerful devices like Macs are widespread but unutilized for large-scale inference, and existing distributed inference systems suffer from heavy communication overhead.
How the pieces fit together
The system has three main layers: Runtime, Communication, and Worker. The Runtime Layer is the core orchestration engine for high-throughput, server-side LLM serving on distributed, heterogeneous networks, composed of an Executor (control loop), Model Shard Holder, Request Manager, Scheduler, and Paged KV Cache Manager.
Underneath, Parallax leans on well-established engines rather than reinventing kernels:
- GPU workers: A modified version of SGLang, which uses PyTorch with CUDA kernels on NVIDIA hardware, with asynchronous batching for heterogeneous compute. vLLM is also supported as a backend.