Aphrodite Engine Runs Every Quantized Model vLLM Refuses to Load
Aphrodite Engine is a vLLM-derived inference server that keeps the throughput but adds nearly every quantization format and sampler community models actually ship with.
PRO- Aphrodite Engine is a vLLM-based inference server powering PygmalionAI and Dolphin Inference Network.
- Supports 15+ quantization formats including AQLM, AWQ, GPTQ, EXL3, GGUF, MXFP4, and VPTQ.
- Runs on NVIDIA CUDA, AMD ROCm, Intel XPU, Google TPU, Apple Metal, and CPU.
- Includes tensor, pipeline, data, and expert parallelism without requiring a Ray cluster.
- Ships speculative decoding via MTP, EAGLE, DSpark, DFlash, and n-gram methods.
- Exposes OpenAI, Anthropic, and Kobold APIs on port 2242 out of the box.
If you have ever tried to serve a random GPTQ or EXL2 model pulled off Hugging Face and hit a wall with the default inference stacks, Aphrodite Engine is worth a serious look. It is a large-scale LLM inference engine that forks vLLM and layers on a wider quantization matrix, extra samplers, more hardware backends, and a broader API surface. Aphrodite currently serves production workloads for the Dolphin Inference Network and PygmalionAI.
vLLM's core, with the community features you actually need
Aphrodite optimizes the serving of Hugging Face-compatible models at scale, building on vLLM's PagedAttention to deliver high-performance inference for many concurrent users. The lineage matters: rather than reinvent the scheduler, the project inherits the paged KV cache and continuous batching from upstream, then aggressively expands the surface area around them.
What sets it apart is scope. The additions map almost exactly to what you need when pulling random checkpoints off Hugging Face, starting with a quantization list that reads like a full survey of the field: AQLM, AutoRound, AWQ, BitNet, Bitsandbytes, ExLlamaV3, GGUF, GPTQ, QuIP#, SqueezeLLM, Marlin, NVIDIA ModelOpt, TorchAO, VPTQ, compressed_tensors, MXFP4, and more.
What ships in the box
The current feature set covers most things a production serving stack needs, and a lot that vLLM alone does not:
- Continuous batching and paged KV-cache management, with prefix caching on by default
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
