Seoul National University's Cluster-aware Upcycling Turns Cloned AI Experts Into Real Specialists

Turning a pretrained dense transformer into a Mixture-of-Experts (MoE) model is one of the cheapest ways to scale capacity without paying the full pretraining bill. The standard trick, called Sparse Upcycling, duplicates the dense feed-forward network into N identical experts and adds a random router on top. It works, but a nagging problem remains: every expert starts as a clone, so they take a long time to actually differentiate.

A team from Seoul National University and LG AI Research has a fix. In a paper accepted to CVPR 2026, Sanghyeok Chu, Pyunghwan Ahn, Gwangmo Song, Seung Hwan Kim, Honglak Lee, and Bohyung Han introduce Cluster-aware Upcycling, an initialization scheme that bakes semantic structure into the experts and the router before training begins. The paper shows the approach beats existing upcycling methods on CLIP ViT-B/32 and ViT-B/16 across zero-shot and few-shot benchmarks.

Why cloned experts are a lousy starting point

Sparse Upcycling initializes an MoE from pretrained dense weights instead of training from scratch. Because all experts start from identical weights and the router is randomly initialized, the model suffers from expert symmetry and limited early specialization. In practice the router has no reason to send a given token to any particular expert, and the experts have no reason to diverge. Training eventually breaks the tie, but a lot of compute burns before the model becomes a real mixture of specialists.

Cluster first, then carve up the weight matrix

The core idea is to give each expert a different slice of the input distribution from the start. The method first partitions the dense model's input activations into semantic clusters, then initializes each expert using the subspace representations of its corresponding cluster via truncated SVD, while setting the router's initial weights to the cluster centroids.