CMU's Event Tensor Compiler Makes AI Inference 1.48x Faster on Qwen3
A new compiler abstraction from CMU, NVIDIA and collaborators fuses entire LLM decoding loops into a single persistent GPU kernel, cutting warmup 3.5x and beating vLLM by up to 1.48x.
Takeaways
- Event Tensor is a new compiler abstraction that encodes GPU task dependencies as symbolic-shaped tensors, enabling dynamic megakernels.
- Handles both dynamic batch shapes and data-dependent MoE routing without runtime recompilation or CUDA Graph recapture.
- ETC compiler supports both static per-SM scheduling and on-GPU dynamic push/pop scheduling as compile-time transformations.
- Achieves up to 1.48x speedup over vLLM and 1.20x over SGLang on Qwen3-30B-A3B batch-1 decoding on B200.
- Warmup time drops from 123s (vLLM) and 583s (SGLang) to 35s on Qwen3-32B, with zero runtime graph captures.
- Integrated into a major open-source system; abstraction is DSL-agnostic and could sit atop Triton or CuteDSL.
Megakernels have become the frontier trick for squeezing latency out of LLM inference. Rather than launching hundreds of small GPU kernels per decoding step, you fuse the whole model into one persistent kernel that never leaves the SMs. The catch is that existing megakernel work only handles fixed shapes and static dataflows, which rules out the two things every real serving system needs: continuous batching and Mixture-of-Experts routing. A large collaboration from CMU, NVIDIA, Berkeley, Princeton, Tsinghua, Peking, Shanghai Jiao Tong and SJSU (led by Hongyi Jin, with Tianqi Chen and Zhihao Jia among the authors) just published Event Tensor, a compiler abstraction that makes dynamic megakernels tractable.
Kernel launches are the new bottleneck
On modern GPUs, each kernel launch typically incurs 5 to 10 microseconds of latency, while the fastest kernels may complete in 2 microseconds. During autoregressive decoding, a single step can fire off hundreds of tiny ops, so launch overhead dominates. CUDA Graphs help by capturing and replaying a fixed sequence, but they preserve kernel boundaries, meaning the implicit synchronization between launches still blocks fine-grained pipelining that could otherwise overlap independent work.
Megakernels take the opposite approach: fuse multiple operators into a single persistent kernel, eliminating launch overheads and enabling inter-operator parallelism. Each operator is chopped into tile-sized tasks distributed across streaming multiprocessors, and lightweight signaling enforces dependencies. Prior systems like Hazy Research's low-latency Llama-1B megakernel and Mirage showed the payoff, but they were built by hand for single-batch dense inference. Anything with a symbolic batch dimension or data-dependent control flow (read: MoE) broke the assumption.