DeepSeek's DeepGEMM Fuses Entire AI Expert Layers Into One GPU Operation
DeepSeek's open-source CUDA kernel library gets Mega MoE, FP8×FP4 GEMM, and a fused MoE mega-kernel that overlaps NVLink comms with tensor core compute
PRO- DeepGEMM, DeepSeek's open-source CUDA kernel library, received a major update adding Mega MoE, FP8×FP4 GEMM, and an FP4 indexer.
- Mega MoE fuses the entire MoE forward pass (dispatch, two linear layers, SwiGLU, combine) into one kernel that overlaps NVLink comms with tensor core compute.
- FP8×FP4 GEMM halves weight memory vs FP8-only and matches the precision regime used in DeepSeek V4 inference.
- The library now officially supports both SM90 (Hopper) and SM100 (Blackwell datacenter) GPUs; SM120 (RTX Pro 6000) is not yet officially supported.
- vLLM and SGLang both detect and use DeepGEMM automatically when installed on FP8-capable hardware.
- The update is MIT-licensed and available now at github.com/deepseek-ai/DeepGEMM; requires PyTorch 2.9+ for Mega MoE.
DeepGEMM, DeepSeek's open-source CUDA kernel library for high-performance matrix multiplication, just received its most substantial update since launch. The April 2026 release lands Mega MoE, FP8×FP4 GEMM, an FP4 lightning indexer, Programmatic Dependent Launch (PDL) support, and a faster JIT compilation path , all under the MIT license.
What DeepGEMM actually is
DeepGEMM is a small CUDA library that takes a few well-chosen FP8 GEMM and MoE shapes seriously, hand-tunes them on top of CuTe and CUTLASS primitives, and packages them behind a JIT-compiled Python API. It is the kernel layer DeepSeek wished existed when shipping V3, R1, V3.2, and now V4 in FP8.
Standard cuBLAS GEMM is tuned for large square matrices. MoE expert layers use grouped GEMM with variable batch dimensions , not all experts get equal token load per step, so the effective (M, N, K) shapes change constantly. A standout feature of DeepGEMM is its use of fine-grained scaling. In the context of LLMs, GEMM operations are the primary computational bottleneck. By applying fine-grained scaling within these kernels, DeepGEMM allows for more precise control over the quantization process, which is vital when working with the limited dynamic range of 8-bit formats.
The April 2026 update, piece by piece
The headline additions are Mega MoE, FP8×FP4 GEMM, an FP4 Indexer, PDL, and faster JIT compilation. Here is what each one means in practice:
- Mega MoE , a fused mega-kernel that collapses the entire MoE forward pass into a single GPU operation. It fuses expert dispatch, the first linear layer, SwiGLU activation, the second linear layer, and expert combine , while overlapping NVLink communication with tensor core computation. Only FP8×FP4 MoE is supported, and it requires PyTorch 2.9 or higher.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
