Ant Group's Humming Open-Sources Faster AI Inference, Hitting 5x Gains

Ant Group and the vLLM community have open-sourced Humming, a component-based CUDA kernel library for quantized inference. It composes reusable kernel components and uses just-in-time compilation to specialize them for each weight format, activation format, GPU architecture, and workload shape.

Quantization lowers model memory use and bandwidth demand, but those savings depend on kernels tuned for the chosen format and hardware. Maintaining separate implementations for every combination consumes engineering time, complicates serving stacks, and leaves less common configurations on slower fallback paths. Humming provides a shared execution layer intended to reduce that duplication.

The matrix behind one quantized model

Formats such as W8A8 and W4A16 specify the bit widths used for weights and activations. W8A8 uses 8-bit weights and activations, while W4A16 combines 4-bit weights with 16-bit activations. WNA16 covers weight-only configurations with N-bit weights and 16-bit activations.

Format names capture only part of the kernel contract. Scale granularity, quantization group size, zero points, symmetric or asymmetric encoding, tensor layout, batch shape, and GPU instruction support all affect implementation. Dedicated kernels for each combination create a rapidly expanding maintenance surface.

One compiler path through the matrix

Humming converts output from different quantization tools into a standard description of weights, scales, zero points, and layouts. Three configuration objects then control compilation:

• Layer Config: Tensor shapes, data types, quantization parameters, and target GPU architecture.
• Compute Config: General matrix multiplication type, batch invariance, and input-scale layout.
• Tuning Config: Tile dimensions, pipeline stages, warp specialization, memory access patterns, and matrix multiply-accumulate instructions.

At the CUDA level, Humming exposes reusable components for task scheduling, data movement, dequantization, Tensor Core computation, and output processing. Adding a format or architecture usually requires a focused component extension while preserving the shared kernel structure. The JIT compiler removes unused branches when it generates the final specialization.

In one W4A16 compilation test on an H200 with CUDA 13.4, Humming compiled a kernel with NVCC in 1.82 seconds, compared with 23.35 seconds for CUTLASS with NVCC. Compiling and loading the supported M-dimension specializations for one layer’s matrix multiplication took approximately two to four seconds. In these workloads, M generally corresponds to the number of token rows processed together.

Coverage reaches beyond standard PTX types

Current project documentation describes full support for NVIDIA architectures from Turing onward, along with T-Head’s Zhenwu 810E as a CUDA-compatible target. Humming also supports two undocumented Blackwell Tensor Core formats, float4e0m3 and float8e3m4, which the PTX type system does not expose. Those paths target SM100, SM103, SM120, and SM121 GPUs.

Pre-Blackwell GPUs lack native MXFP4 Tensor Core support, so scaling and dequantization can constrain W4A8-MXFP4 performance at larger batch dimensions. Humming converts MXFP4 weights to INT8 or FP8 inside the kernel, allowing an H200 to retain 4-bit weight storage while computing through formats supported by its Tensor Cores.

Published gains vary by workload

The project’s published results report the following performance ranges:

• Dense W4A16: Humming matches or exceeds vLLM Marlin across most tested shapes, with reported gains of approximately 3 to 4 times at several small-batch configurations. Large-batch results reach about 2 times on H200 and more than 3 times in some B300 cases.
• MoE W4A16: MXFP4 tests on H20 and H200 reach roughly 2 times Marlin performance in several medium and large batches. On B300, MXINT4 and NVFP4 show larger gains at high batch sizes, with one grouped-execution result approaching 5 times its listed baseline.
• FP8 W8A8: Hopper results reach approximately 2.5 to 2.6 times DeepGEMM Contiguous at small and medium batch sizes.
• W4A8-MXFP4: The H200 path reaches about 4.5 times the published DeepGEMM W8A8 baseline.

Mixture-of-experts models route tokens among multiple expert matrices, making grouped matrix multiplication and irregular batch shapes central to performance. The published comparisons also cover different precisions and storage formats, so some results measure a combined format-and-kernel change rather than an isolated backend improvement. Exact gains will depend on model shape, routing distribution, batch size, and GPU.

Serving gains include an activation change

An end-to-end DeepSeek-V4.1-Flash-1017 test used eight H20 GPUs with tensor parallelism set to eight. Replacing Marlin W4A16 MoE execution with Humming W4A8 produced the following reported changes. Concurrency is the number of simultaneous requests, TTFT is time to first token, and TPOT is mean time per output token.

At concurrency 64, reported output throughput increased from 442.37 to 633.94 tokens per second, while mean TPOT fell by 30.5%. The test retained the same 4-bit weight checkpoint and changed the activation path from 16-bit to 8-bit alongside the kernel backend. As a result, the measurement captures both changes. The cited results do not include model-quality measurements, which deployments should validate for their own prompts and evaluation sets.

Install, warm up, serve

The standalone package is available through a CUDA 13 wheel that the project reports at approximately 500 KB. Humming compiles kernels on demand, avoiding a distribution containing gigabytes of precompiled specializations.

pip install 'humming-kernels[cu13]'

vllm serve deepseek-ai/DeepSeek-V4.1-Flash-1017 \
    --tensor-parallel-size 8 \
    --linear-backend humming \
    --moe-backend humming

CUDA installations of vLLM 0.29.0 and later include Humming as a dependency, removing the separate installation step. The --quantization humming compatibility path can load formats that vLLM does not handle natively, including GPTQ INT3.

On-demand compilation shifts some work into startup and warmup. Operators should measure cold-start time across their actual layer shapes and account for compilation in readiness checks, autoscaling policies, and deployment rollouts.

Strong fit, unfinished autotuning

Humming currently fits workloads that combine mixture-of-experts routing, aggressive quantization, and several GPU generations. Its Hopper W4A8 path is especially relevant to MXFP4 checkpoints that otherwise execute with BF16 activations through W4A16 kernels.

• No benchmarking-based autotuner: Model shapes, quantization settings, and GPU resources make configuration selection difficult, and the current release cannot guarantee the fastest tuning choice for every case. The project plans to add benchmark-driven autotuning.
• Different Rubin exposure: Rubin does not expose the hidden Blackwell Tensor Core formats, so those native paths do not transfer to that architecture.
• Incomplete instruction coverage: INT8 UMMA on B300 does not yet run at full performance and falls back to slower matrix multiply-accumulate paths.

One integration surface, several checks

Teams currently maintaining separate Marlin, FlashInfer, and DeepGEMM adapters can use Humming as a common format and compilation layer. Related work already includes a Humming-compatible Chord integration, while FlashInfer’s CUTLASS W4A8 implementation builds on the same approach.

Adoption should be evaluated against each deployment’s exact model shapes, quality requirements, startup budget, and latency targets. Humming’s main engineering proposition is a smaller kernel-maintenance surface paired with per-shape specialization across quantization formats and GPU generations.