HyperQwen Runs Qwen3.8-27B on a Single RTX 3090 at 1,035 tok/s
HyperQwen crams a 27B Qwen model onto one 24GB RTX 3090 with vLLM patches, hitting 127 tok/s solo and ~1,035 tok/s at 64 concurrent.
- HyperQwen serves Qwen3.8-27B on a single 24GB RTX 3090 via patched vLLM.
- Hits 127 tok/s single-user, 381 tok/s when reproducing prompt content, 1,035 tok/s at 64 concurrent.
- Supports 150k-262k context using the KVarN 4/2-bit KV cache backend.
- Prefix cache cuts TTFT from 22s to 0.56s on repeated questions over the same document.
- Two Docker Compose profiles: single-user chat mode and high-throughput batch mode.
- Apache 2.0, OpenAI-compatible API, with 38-file vLLM patch series documented.
HyperQwen serves Qwen3.8-27B from one RTX 3090
A 27-billion-parameter model strains a 24 GB card because its weights, temporary activations, and key-value cache compete for memory. The HyperQwen repository packages a patched vLLM stack and a requantization pipeline that converts the Qwen3.8-27B checkpoint into lower-precision artifacts for single-GPU serving.
The headline figures come from project benchmarks on an RTX 3090 capped at 250 watts. Performance depends heavily on concurrency, context length, and how much of the output can be drafted from the prompt, so broader claims require independent testing across other cards and checkpoints.
Five profiles follow the traffic
| Profile | Configuration | Best fit | Reported result |
|---|---|---|---|
| A: Batch | Default | Concurrent API traffic | About 1,035 tok/s aggregate decode at 64 concurrent requests |
| B: Single default | Default | One active user | 127 tok/s with a 64k context |
| C: Reproduction | DFLASH_TOKENS=15 |
Extraction, quoting, and prompt-grounded transformations | 381 tok/s while reproducing prompt text, with a 56k context |
| D: Long context | SPEC=mtp CTX=long |
Large prompts with faster generation | About 95 to 100 tok/s with a 150k context |
| E: Huge context | CTX=huge |
Maximum packaged context | 67 tok/s on mixed generation with a 240k context |
The KVarN cache backend reaches a reported maximum of 262k tokens in separate testing, while profile E uses a 240k setting. Ordinary open-ended chat runs at about 133 tok/s in the repository’s measurements.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
