EXO Labs Builds the Missing Benchmark Site for Running LLMs on Consumer Hardware

EXO Labs , the team behind the open-source distributed inference framework that lets you chain together Macs and workstations into a local AI cluster , just teased its next big move: a comprehensive, free benchmarking website covering every major model, every quantization level, and every consumer hardware configuration across every price point. The site, benchmarks.exolabs.net, is live but currently shows "Coming soon."

The question everyone keeps asking

The most common question EXO Labs gets is about benchmarking , things like "How many tokens per second can I get if I connect 3 Macs and run Llama 70B?" Right now, answering that requires either running the experiment yourself or piecing together scattered community results. EXO wants to make that a one-stop lookup.

The plan is to launch a free benchmarking website providing detailed hardware configuration comparisons, helping users choose the best LLM operating solution based on their needs and budget. The tweet makes clear the ambition is broad: every model, every quant, every hardware setup, every price point.

What EXO actually is

EXO is an open-source framework for running large language models efficiently across mixed hardware setups. Rather than treating inference as a task bound to a single GPU or accelerator, EXO automatically spreads workloads across whatever devices you have , turning a cluster of desktops, laptops, workstations, servers, tablets, or even smartphones into a cooperative AI mesh.

At its core, EXO uses pipeline parallel inference , it takes an LLM and splits it into multiple "shards" (contiguous slices of the model's layers), assigning each shard to a different device. Devices can be different accelerators on the same machine like a GPU, CPU, or NPU, or physically separate machines connected over a network. The open-source repo has accumulated over 44,500 GitHub stars.

What the benchmarks already show

EXO ran an earlier round of benchmarks on M4 Pro clusters, and the results reveal something counterintuitive about distributed inference. Here's what they found with LLaMA 3.2 3B:

Adding more devices actually decreases single-request performance due to network overhead, but multi-request throughput scales nearly linearly , 2.2x for 3 devices in their tests. Pipeline parallel inference is most beneficial for batch workloads and parallel inference scenarios, and network latency (not bandwidth) is typically the bottleneck for inter-device communication.

EXO clustering is the only path to 70B-class inference on Mac mini and MacBook hardware without a Mac Studio. And the team has demonstrated 671B parameter models running across Mac Mini clusters.

The real gap this fills

The local inference space is growing fast, but it has a serious information problem. Quantization level (how aggressively you compress a model's weights to fit in less memory), batch size, and prompt length each swing performance by 2-5x. Community benchmarks are scattered, often outdated, and rarely cover mixed-hardware setups. There is no authoritative, hardware-specific reference that tells you what you'll actually get before you spend money on a machine.

EXO is continuously expanding its benchmarking suite to include more heterogeneous setups , mixing different devices and even different model quantizations. The new site appears to formalize and scale that effort significantly, with the teaser image showing hardware like the NVIDIA DGX Spark already in the pipeline.

Who this is for

• Hardware shoppers: Deciding between a Mac Studio, a Mac Mini cluster, or an NVIDIA DGX Spark? You'll be able to compare real token-per-second numbers at your budget.
• Teams self-hosting models: Running LLMs via managed services is convenient, but costs can spiral quickly once you scale to many queries , this gives you the data to justify the switch to local.
• Researchers and developers: Quickly identify which quantization of a given model hits the right speed/quality tradeoff on the hardware you already own.
• Batch workload operators: Batch workloads like summarizing a set of documents don't need fast individual requests , they need to process the batch in the least amount of time, which depends on total throughput. Knowing multi-request TPS across cluster sizes is exactly the number you need.

What's still unknown

The site is not live yet, and the tweet says "releasing soon" without a firm date. It's also unclear how frequently benchmarks will be updated as new models drop, or whether the suite will cover Windows and Linux GPU setups beyond Apple Silicon. EXO is still working on heterogeneous setups, which require a more sophisticated scheduler to ensure device utilization stays high.

That said, the direction is clear: EXO Labs is positioning itself as the authoritative source for local inference performance data , the equivalent of a GPU benchmark database, but for LLMs on consumer hardware. If the execution matches the ambition, it could become a standard reference every time someone asks "what should I buy to run X model?"