Skip to main content
Back

Gradient's Parallax Turns Idle Macs and Gaming PCs Into a 235B AI Cluster

Gradient's open-source Parallax turns a pile of mismatched Macs and GPUs into a single inference cluster, hitting 3.1x throughput over Petals.

Pro

Repo, Full Access
  • Gradient released Parallax, an Apache-2.0 distributed inference engine for heterogeneous clusters.
  • Shards LLMs via pipeline parallelism across NVIDIA GPUs and Apple Silicon using SGLang, vLLM, and MLX backends.
  • Benchmarks show 3.1x throughput and 5.3x lower inter-token latency versus Petals on Qwen 72B/235B.
  • Supports 40+ models including DeepSeek V3.2, Kimi-K2, Qwen3-Next, GLM-5, gpt-oss, and Llama 3.
  • P2P layer built on Hivemind DHT; nodes join a Swarm via parallax run and parallax join.
  • Read the technical paper for scheduler and paged KV cache internals.

Renting a datacenter GPU to serve your own LLM is expensive, and most of us have laptops, gaming rigs, and Mac Studios sitting mostly idle. Parallax, an open-source project from Gradient, tries to close that gap by letting you stitch those machines together into a single distributed inference cluster, regardless of whether they run CUDA or Apple Silicon.

It bills itself as a fully decentralized inference engine for building your own AI cluster across nodes with varying configurations and physical locations. The project is Apache 2.0, and the code, docs, and a technical paper are all public.

Shard the model, not your budget

Rather than loading a 70B model onto one big GPU, Parallax splits it across a mesh of peers and streams activations between them. The system runs on a distributed architecture called the Swarm, where your prompt is tokenized and executed across peers. The motivation is blunt: most researchers and developers cannot afford large GPU clusters, powerful devices like Macs are widespread but unutilized for large-scale inference, and existing distributed inference systems suffer from heavy communication overhead.

How the pieces fit together

The system has three main layers: Runtime, Communication, and Worker. The Runtime Layer is the core orchestration engine for high-throughput, server-side LLM serving on distributed, heterogeneous networks, composed of an Executor (control loop), Model Shard Holder, Request Manager, Scheduler, and Paged KV Cache Manager.

Underneath, Parallax leans on well-established engines rather than reinventing kernels:

  • GPU workers: A modified version of SGLang, which uses PyTorch with CUDA kernels on NVIDIA hardware, with asynchronous batching for heterogeneous compute. vLLM is also supported as a backend.