Perplexity's Lily Beats MLX-LM by 1.35x Running Qwen3.6 on Apple Silicon

Perplexity has released Lily, a local inference engine written from scratch for Apple silicon and hand-tuned for a single model: Qwen3.6-35B-A3B. It powers the on-device half of Perplexity Computer's hybrid setup, where cloud models handle heavy reasoning while a Mac-resident model works with private files and apps.

Rather than building a general framework like MLX-LM, the team compiled the model's exact structure, quantization scheme, and expert routing directly into a Rust runtime with custom Metal kernels, cutting PyTorch and MLX out of the execution path entirely.

What Lily actually does

Lily is a single-process engine that loads the model checkpoint, manages session state and the generation loop, and exposes an OpenAI-compatible chat-completions API that streams tokens. Custom Metal kernels handle all Qwen-specific operations. The target checkpoint is Qwen3.6-35B-A3B, a sparse mixture-of-experts model with 35 billion parameters that activates roughly 3 billion per token, using a router that picks 8 of 256 experts plus one shared expert per token.

The model architecture is also unusual: it mixes 10 full-attention layers with 30 Gated DeltaNet layers, a linear-attention variant that maintains a fixed-size recurrent state. That combination drove most of Lily's design decisions.

The numbers

Benchmarked on an M5 Max MacBook Pro with a 40-core GPU and 128 GB unified memory, Lily averages 1.23x MLX-LM's prefill throughput and 1.35x its decode throughput across prompt and context lengths from 256 to 128K tokens. At a 4K prompt with a 4K decode context, the engine reaches 5,749.9 prefill tokens per second and 186.6 decode tokens per second, against 4,737.5 and 140.9 for MLX-LM.

Output quality holds up under a teacher-forced comparison: Lily's perplexity was only 0.04% higher, and it selected the same top-ranked token at 96.35% of tested positions.

Why prefill and decode get different treatment

Prefill processes the prompt in bulk and can reuse each weight block across many token rows, making it a compute-bound matrix problem. Decode generates one token at a time in batch-1 mode, forcing another full pass over the weights per token and shifting the bottleneck to memory bandwidth. Apple silicon exposes different hardware paths for each: compatible GEMMs can use the Neural Accelerator in each GPU core through Metal 4 tensor operations, while batch-1 decode uses matrix-vector kernels that lean on the GPU's vector ALUs instead.

Prefill optimizations

The 35B checkpoint is stored as groupwise 4-bit weights, roughly 19.4 GB on disk. Two changes drove most of the prefill gains:

• Fused dequantization inside the grouped GEMM. Rather than expanding 4-bit weights into a full bfloat16 array in unified memory, Lily unpacks one tile at a time into threadgroup memory just before multiplication. At a 512-token prompt, this increased end-to-end prefill throughput by 77.4%.
• GPU-resident expert routing. The histogram, prefix scan, scatter, and block-map build for MoE routing all stay in a single command buffer. At a 512-token prompt, keeping routing on-GPU increased end-to-end prefill by 89%.
• Register-resident DeltaNet scan. Each simdgroup owns a column of the recurrent state and carries it through the entire scan in registers, avoiding repeated 256 MiB spills per layer.

Decode optimizations

Token handoff between steps stays on the GPU using two alternating command buffers and two token slots, eliminating the CPU round-trip per generated token. Kernel launches are also restructured: in one recorded batch-1 decode step, generating a token launched 795 GPU kernels whose dependencies formed 555 sequential stages, with some kernels free to run concurrently. Metal's default serial execution ignores that, so Lily records the actual dependency graph in a concurrent Metal pass and lets independent kernels overlap.

GQA packing is the biggest single lever for attention. Because eight query heads share one KV head, packing four query heads into a threadgroup lets them share KV loads. Against the unpacked baseline, this improved end-to-end decode throughput by 23.8% at a 32K-token context. At longer contexts the engine switches attention layouts, improving end-to-end decode by 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K.

What failed

The writeup is candid about dead ends. Speculative decoding, which uses a smaller model to propose tokens for the full model to verify, made batch-1 decode 18% slower: verification batches of 2-5 rows are an awkward shape, and different rows fetched different experts. Broader kernel fusion, larger prefill tiles, and router acceleration also failed to move the end-to-end number. The engine is already close to hardware limits on the hot paths, with MoE GEMMs and GEMVs reaching 97.9% and 90.3% of the fastest sustained weight-read rates for their access patterns.

Who this is for

Lily targets a narrow use case. It pays off if you are running Qwen3.6-35B-A3B locally on an Apple silicon Mac with enough unified memory for the 19.4 GB Q4 checkpoint plus KV cache, building agentic or multi-turn workloads where local latency compounds across many tool calls, or studying how to fit MoE and hybrid attention/linear-recurrence models onto consumer hardware. Swap in any other model, quantization scheme, or chip and the specialization dissolves. As open-weight frontier models grow architecturally stranger and consumer chips more heterogeneous, a single generic runtime cannot stay optimal across all workloads. Lily is a concrete demonstration that model-plus-hardware co-design belongs in the local inference stack, not only in datacenter serving.