---
title: "Microsoft's Dion3 Makes the Optimizer Powering Kimi K2 Run 6x Faster"
subtitle: "Microsoft Research's Dion3 optimizer slashes Muon's orthogonalization overhead by up to 6x through smarter math, custom kernels, and selective row updates."
url: "https://alphasignal.ai/news/microsoft-s-dion3-makes-the-optimizer-powering-kimi-k2-run-6x-faster"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-08-12T00:00:00.000Z"
updated: "2026-08-22T02:39:42.216Z"
type: "Paper"
topics:
  - "Training Infra"
  - "Gpus"
subtopics:
  - "Distributed Training"
  - "Scaling Laws"
  - "Kernels"
upvotes: 367
source_url: "https://arxiv.org/abs/2608.11612"
comments: 0
truncated: true
---

# Microsoft's Dion3 Makes the Optimizer Powering Kimi K2 Run 6x Faster

> Microsoft Research's Dion3 optimizer slashes Muon's orthogonalization overhead by up to 6x through smarter math, custom kernels, and selective row updates.

Muon has quietly become the optimizer powering some of the largest language models being trained today, from Kimi K2 to GLM-5\. It comes with a tax, though: an expensive Newton-Schulz orthogonalization step that gets painful as models grow. A team from Microsoft Research, Princeton, NYU, and NVIDIA just released [Dion3](https://arxiv.org/abs/2608.11612), a drop-in Muon replacement that matches or improves on the loss Muon achieves while reducing optimizer step time by up to 6x. It ships today as part of the [microsoft/dion](https://github.com/microsoft/dion) package.

![Bar chart showing optimization step time reduced from 26x AdamW to 4x AdamW across Dion3's contributions](https://arxiv.org/html/2608.11612v1/x1.png) 

## Why Muon needs a rescue

Muon works by taking the momentum matrix and running it through a polar decomposition, which rebalances the singular values of each update. It reaches a given loss in fewer steps than AdamW, but each step is heavier. The overhead comes from Muon's Newton-Schulz orthogonalization procedure, a cubic-time matrix operation not present in older optimizers. When weights are sharded across GPUs, Muon must gather the shards together to compute polar on the full matrix, piling on communication overhead that AdamW doesn't have.

Kimi's team made Muon scale, but the authors point out that success required a very specific alignment of architecture, parallelism strategy, and framework versions. Dion3 is designed to remove that fragility.

## Four ideas that stack

Dion3 attacks the overhead at every layer of the stack. Each of the four contributions is useful alone, and they compound when combined:

1. **Gram Newton-Schulz.** A mathematically equivalent reformulation that iterates on the small n by n Gram matrix instead of the full n by m weight matrix.
2. **Custom CuteDSL symmetric GEMM kernels.** Exploit the symmetry of intermediate matrices to halve FLOPs on Hopper and Blackwell GPUs.
3. **Fractional update rule.**

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Dion3: Full-Stack Orthogonal Updates](https://arxiv.org/html/2608.11612)
- [microsoft/dion GitHub repository](https://github.com/microsoft/dion)
- [Dion: Distributed Orthonormalized Updates](https://arxiv.org/abs/2504.05295)
- [Muon optimizer](https://kellerjordan.github.io/posts/muon/)
