Generated Computers' Locals Triples AI Agent Throughput on Apple Silicon
Locals is an open-source local inference server for Apple Silicon built around multi-agent workloads, hitting roughly 3x the throughput of mlx-lm on Macs.
- Locals is a new Apache 2.0 local inference server for Apple Silicon, built for agents not chat.
- Hits up to 3x mlx-lm throughput with 8 concurrent requests on Qwen3.5-4B.
- Based on the paper Breaking the Space Barrier, moving grammar checks onto the GPU.
- Works with OpenCode, Cline, Aider, OpenHands, Claude Code via one-command launchers.
- Ships kernels for Qwen3.5, LFM3, MiniCPM5, Gemma 4, and Recursive's 40B Flint MoE.
- OpenAI, Anthropic and Ollama API compatible on port 8321; install via
curloruv.
Generated Computers has released Locals, an open-source inference server designed to run concurrent coding agents on Apple silicon. It batches parallel requests, caches repeated prompt prefixes, and applies tool-call grammars on the GPU. In maintainer-published tests, those optimizations deliver up to three times the throughput of mlx-lm at eight concurrent requests.
Coding-agent traffic combines long prompts, repeated context, structured tool calls, and several simultaneous workers. A planner, coder, and reviewer may share most of a prompt while generating separate responses. Locals organizes inference around that workload, allowing the M-series GPU to process small batches and reuse conversation state.
Concurrency opens the gap
The headline benchmark runs Qwen3.5-4B on an M2 Pro with 16 GB of memory. mlx-lm reaches a reported aggregate 87 output tokens per second at both four and eight concurrent requests. Locals continues scaling and reaches roughly three times that throughput at eight requests. Performance at concurrency one remains close.
| Workload | Locals | Comparison |
|---|---|---|
| Eight agents, time to first token | 1.6 seconds | Ollama: 42 seconds |
| One agent rewriting a 100-line file | 128 tokens/s | mlx-lm: 43 tokens/s |
| Follow-up on a 48,000-token document | Generation begins in 0.6 seconds | mlx-lm: out of memory |
| Sixteen tool-calling agents | 29 seconds | mlx-lm: 75 seconds |
All figures come from the project maintainers and await independent reproduction. Model builds, quantization, prompt length, output length, sampling settings, hardware temperature, and cache state can materially change inference results.
Grammar work moves to the GPU
Constrained decoding limits the model’s available next tokens so its output follows a JSON schema or tool-call format. Locals grounds its implementation in the paper
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
