Cursor's Mixture-of-Kittens Open-Source Kernel Trains AI 2.37x Faster

Mixture-of-Kittens (MoK) is Cursor's new open-source CUDA megakernel for training Mixture-of-Experts (MoE) models on NVIDIA GB300 NVL72 racks. Already running in production to train Composer, Cursor's agentic coding model, across tens of thousands of GPUs, it benchmarks up to 2.37x faster on MXFP8 forward passes and delivered a 1.41x end-to-end training throughput gain over Cursor's previous DeepEP-based stack.

Where the time was actually going

In large MoE models, each input token is routed to a small subset of "experts" (specialized feed-forward sub-networks) that may live on different GPUs. Tokens travel across the network before and after computation in a process called dispatch and combine. Depending on the workload, the MoE layer can consume more than half of total training time. The compute portion had already been optimized; the bottleneck had shifted to inter-GPU communication.

Earlier Cursor work, including custom MXFP8 and NVFP4 training kernels and a "warp decode" approach for MoE inference, addressed only the compute side and left inter-GPU communication to separate systems. In production, communication had become the limiting factor and forced a ground-up redesign.

Moving to GB300 NVL72s added two new constraints. An NVL72 is a multi-node rack within a single NVLink domain, enabling fine-grained overlap of computation and communication across all 72 GPUs. But the integrated Grace CPUs are slow relative to the GPUs, so GPU streams quickly caught up to CPU-side work, leaving the GPU idle. Any solution had to minimize CPU involvement aggressively.

What MoK actually does

MoK fuses all MoE communication and computation into a single persistent kernel. There are no kernel launch boundaries separating dispatch, expert computation, and combine. Everything runs inside one kernel, with different groups of Streaming Multiprocessors (SMs) assigned to different tasks and coordinating through shared counters. The design is also fully deterministic: the same input produces bitwise-identical output regardless of hardware scheduling.

MoK targets DeepSeek-V3-style MoE layers, used across open-weight models including GLM, Qwen, Kimi (up to K2.7), and DSV itself. Training or fine-tuning any of those architectures at scale makes MoK directly relevant.

Four engineering decisions behind the gains

1. Pull-based dispatch. Most existing systems, including DeepEP, use push-based communication where the sending GPU writes tokens directly into remote GPU memory. MoK uses pull-based dispatch instead, which delivers up to 29% higher NVLink bandwidth utilization under expert imbalance. NVLink has separate lanes for each direction, and pull traffic naturally splits metadata and payload across both lanes, keeping both busy simultaneously.

2. Eliminating cross-GPU signaling. With push-based dispatch, a rank must wait for signals from up to 71 peers and flush memory across the rack with a memory fence. Pull-based dispatch requires no cross-GPU signaling at all. In multi-node microbenchmarks, push-based signaling incurs roughly 5.8x higher latency than pull-based: 103 µs versus 18 µs. That cost accumulates quickly inside a megakernel.

3. Tunable minibatch granularity. The overlap between communication and computation depends on how many tokens transfer per round. Too few and tensor cores never fully saturate. Too many and tensor cores sit idle waiting for the first batch. MoK makes minibatch size a tunable parameter, easy to sweep before running a workload. For Kimi 2.5 shapes on Blackwell, the optimum lands around 2,368 tokens per transfer.

4. Ring token buffers. MoE routing is dynamic, so you cannot know in advance how many tokens will arrive at each GPU. The two standard solutions are token dropping (bad for quality) and CPU-GPU synchronization (bad for throughput, especially on GB300). MoK uses a fixed-size ring buffer called a macrobatch, a few hundred megabytes cycling at minibatch granularity, which eliminates CPU involvement entirely without dropping tokens.

Determinism as a design requirement

MoK fixes the order of floating-point operations so the same input always produces bitwise-identical output. For on-policy reinforcement learning post-training, where reproducible rollouts are required to run clean ablations, this matters considerably. Most high-performance kernels trade determinism for speed; MoK keeps both.

MoK supports BF16 and MXFP8 precision. In MXFP8 mode, the shared expert stays in BF16 for training stability. MXFP8 quantization is fused directly into the dispatch all-to-all and the grouped GEMMs to avoid extra memory traffic.

Benchmark results

Cursor benchmarked MoK against four baselines on GB300 NVL72 hardware with EP degree 64 and 2,048 tokens per GPU, using shapes from Kimi K2.7, GLM-5.2, Qwen3.5-397B, and DeepSeek-V4-Pro:

• Up to 2.37x faster for MXFP8 forward pass
• Up to 1.78x faster for MXFP8 backward pass
• Up to 1.92x faster for BF16 forward pass
• Up to 1.58x faster for BF16 backward pass

In production across 512 GPUs on several GB300 NVL72 racks, MoK delivers 1,070.2 tokens per second per GPU versus 760.9 for the previous DeepEP-based stack, a 1.41x improvement in end-to-end training throughput.

Agents wrote part of this kernel

The MoK team credits coding agents with accelerating the work, particularly on simpler sub-kernels, which freed engineers to focus on the harder distributed systems problems. A year ago, writing a megakernel required building a layer of simplified abstractions first. The team skipped that layer entirely, working through the complexity from scratch with agent assistance. That reflects a real shift in how frontier kernel engineering gets done.

Who should use it

MoK targets GB300 NVL72 racks specifically. Its optimizations are built around NVLink 5th-gen fabric and the Blackwell GPU architecture, so it does not apply directly to other hardware. The underlying design principles, pull-based dispatch, tunable minibatch overlap, ring buffers for dynamic routing, and full kernel fusion, are transferable to anyone building MoE training infrastructure on other platforms.

The practical use cases:

• Training or fine-tuning large MoE models (DeepSeek-V3-style architectures) on NVL72 racks
• Replacing DeepEP or HybridEP+Megatron as the MoE communication layer
• On-policy RL post-training where bitwise determinism is required
• Research into MoE kernel design, since all benchmark code is released alongside the kernel

MoK is fully open-source. The GitHub repository includes the full kernel source, benchmark code, and tuning utilities. Cursor plans to maintain it and welcomes contributions.