Skip to main content
Back

Lucebox Rewrites AI Inference Software, Hits 207 tok/s on a 5-Year-Old GPU

Lucebox hand-tunes CUDA kernels for consumer GPUs, hitting 207 tok/s on Qwen3.5-27B on a single RTX 3090 — no H100 required

Repo, Full Access
  • Hand-tuned CUDA kernels for RTX 3090 push Qwen3.5-27B to 207 tok/s — 3.43x over autoregressive baseline, 2.8x over SGLang AWQ.
  • Megakernel fuses all 24 layers of Qwen 3.5-0.8B into a single CUDA dispatch, matching Apple M5 Max efficiency (1.87 tok/J) at 1.8x the throughput.
  • DFlash + DDTree speculative decoding ported to GGUF (Q4_K_M), enabling 128K context on a 24 GB card — the first public path to do so.
  • PFlash cuts time-to-first-token at 128K from ~257s to 24.8s (10x) via speculative prefill compression.
  • Luce Spark fits 33-35B MoE models on 16 GB GPUs by keeping only active experts resident, with 1.5x speedup over naive offloading.
  • Full stack is MIT-licensed on GitHub, requires Ampere+ GPU + CUDA 12+, with AMD HIP support now available.

The standard assumption in local AI inference is that you pick your poison: fast but power-hungry NVIDIA, or efficient but slower Apple Silicon. Lucebox just broke that framing entirely , by rewriting the software, not upgrading the hardware. On a five-year-old RTX 3090, their hand-tuned kernel stack hits 207 tok/s on Qwen3.5-27B and matches Apple M5 Max efficiency on a smaller model, at nearly twice the throughput. The GPU was never the bottleneck. The frameworks were.

The gap nobody was filling

Consumer GPUs , especially the RTX 3090 with its 24 GB of VRAM and mature CUDA support , are the most common serious AI workstation hardware outside of data centers. Yet almost all kernel-level optimization work targets H100 fleets. The big inference frameworks (vLLM, SGLang, llama.cpp) are built to be general-purpose: decent on everything, great on nothing specific.

Lucebox's thesis is the opposite: pick one chip, one model family, and rewrite everything from scratch for that exact target. The result is a collection of self-contained releases, each with full benchmarks and paper-style writeups, published under MIT license.

Project one: the megakernel

The first release targets Qwen 3.5-0.8B, a hybrid model that alternates between 18 DeltaNet layers (a recurrent, linear-attention mechanism that scales linearly with context length) and 6 standard attention layers. No one had written a fused kernel for this architecture before.

The core insight: standard frameworks run roughly 100 separate CUDA kernel launches per generated token , one per layer operation. Each launch hands control back to the CPU, re-fetches weights, and re-synchronizes threads. Those round-trips are pure overhead. Lucebox's megakernel collapses all 24 layers into a single CUDA dispatch