Tsinghua's MimicLM Beats Top Voice Cloning Systems by Flipping Synthetic Training Data

Voice imitation systems have long faced a data problem: to teach a model how to turn one person's speech into another's while keeping the words identical, you need paired recordings that don't naturally exist. A new paper from Tsinghua University and the Chinese University of Hong Kong, Shenzhen proposes MimicLM, which sidesteps this bottleneck with a clever inversion: use synthetic speech as the input the model learns to transform, and real human recordings as the output it learns to produce.

The resulting architecture is simpler than existing disentanglement-based systems, and human raters gave it the top naturalness score against strong baselines like Vevo and SeedVC v2.

Why parallel voice data barely exists

Voice imitation is more ambitious than plain voice conversion. Traditional voice conversion transforms only speaker timbre while preserving content and prosody, whereas imitation reproduces both timbre and speaking style. To train it cleanly, you'd want triplets where source and target audio share identical words but differ in speaker identity, style, and prosody. Those recordings simply aren't sitting around in any large corpus.

Two workarounds have dominated the field. The first uses architectures that try to explicitly separate content, timbre, and prosody through information bottlenecks and multi-stage training, which gets complicated fast. The second generates synthetic pairs with an external TTS or voice-conversion system, then treats the synthetic output as the target the model should reproduce. That second approach hits a ceiling: since the model learns to reproduce synthetic targets, output quality is inherently bounded by the external system's capabilities.