Tencent's AuK Replaces 16 Speech Tools With One Open-Source Model

Tencent's Hunyuan lab has released AuK, an open-source foundation model that pushes speech work into the same territory image models like Nano Banana occupy: describe what you want in plain language, optionally hand it a reference clip, and the model figures out the rest. Text-to-speech, content rewrites, denoising, source separation, accent removal, pitch shifting, and whisper-to-normal conversion all live behind one call. Code, weights, a Gradio demo, and a ComfyUI node pack are up under an MIT license.

One interface, sixteen jobs

AuK is a 1.5B foundation model for speech generation and editing, trained on millions of hours of audio, and it exposes zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a single natural-language instruction interface. Every task uses the same message format: an instruction string, and optionally a source or reference audio file. The model reads the wording to decide whether you want to denoise, replace a phrase, change the emotion, or synthesize a new line in someone's voice.

The task list is unusually broad for a single checkpoint:

• Generation: zero-shot voice cloning TTS, and instruct TTS driven only by a voice description with no reference audio.
• Content editing: replace, insert, or remove words in an existing recording, including rewriting lyrics while preserving the melody and singer.
• Acoustic editing: pitch shift by semitones, speed change, and volume in decibels.
• Paralinguistic editing: change emotion, timbre, remove regional accents, add or strip breaths and laughs, and convert between whispered and normal speech.
• Restoration: denoising, dereverberation, speech separation by talking order, music vocal extraction, and target-speaker extraction keyed to what a speaker actually said.

How it is put together

AuK combines a multimodal large language model for semantic conditioning, a VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that runs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. In practice, the instruction and any reference audio are read by a Qwen2.5-Omni-3B encoder, the target audio lives in a shared latent from a joint speech-audio-music VAE, and a diffusion-style transformer does the actual synthesis. Rectified flow is the training objective that lets the model draw samples along straight probability paths, which is what makes the distilled variant usable in only a handful of steps.

To support this capability, the team constructed roughly 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision across five task families. Training proceeds in stages: a generation-only warm-up, then joint generation-and-editing pretraining, then two post-training passes, human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation.

Four steps instead of forty

Alongside the base model, Tencent shipped AuK-Flash, a distilled version aimed at latency-sensitive use. The team distilled the model with consistency initialization and task-routed Decoupled DMD, and the resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5x wall-clock speedup over the full model under matched conditions. DMD here refers to distribution matching distillation, a technique for collapsing many diffusion steps into a few while keeping sample quality close to the teacher's. Switching between the two is a checkpoint flag; the Gradio server can even load both on separate GPUs and expose a selector.

Using it

Installation is a standard uv or conda flow with a Python 3.10 environment, plus three checkpoint downloads: AuK, optionally AuK-Flash, and the Qwen2.5-Omni-3B encoder. Every task is one call to auk-infer with an --instruction and optionally an --audio. A content edit looks like this:

auk-infer \
    --audio content.wav \
    --instruction "Replace 'but accepting what we cannot have' with 'and living well with dreams unmet'." \
    --output out_content_edit.wav \
    --gen_seconds 7.0

The Python API mirrors it with a chat-style message list where each turn carries text and audio parts, which will feel familiar to anyone who has worked with multimodal chat APIs:

from auk.infer.infer_auk import AukInfer, save_audio
engine = AukInfer("ckpts/AuK/config.yaml", "ckpts/AuK/auk_base.safetensors")
messages = [{"role": "user", "content": [
    {"type": "text", "text": "Say this in the reference voice: 'Ladies and gentlemen...'"},
    {"type": "audio", "audio": "ref.wav"},
]}]
audio, sr = engine.generate(messages, gen_seconds=6.0)
save_audio(audio, sr, "out.wav")

A Prompt Enhancer utility takes a free-form user request, calls an OpenAI-compatible LLM, and prints a ready-to-run command with the right instruction template, duration, and preprocessing. Fine-tuning uses a JSONL format where each line is a user-assistant pair with instruction, source audio, and target audio, driven through scripts/train.sh.

Where it stands

The authors report leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. That last hedge matters: purpose-built denoisers and separators still have an edge on pure SNR metrics, and AuK is trading a bit of that for the ability to describe restoration jobs in natural language alongside creative edits. The paper hit #2 on Hugging Face's daily papers with 186 upvotes, and the model card is already listed on the tencent/AuK space for a browser demo.

Why it matters

Speech tooling has been fragmenting for years: one model for TTS, another for voice conversion, a third for denoising, a fourth for source separation, each with its own preprocessing quirks. AuK is the argument that a single instruction-conditioned generator, trained on enough varied audio, can absorb that whole pipeline the way image diffusion models absorbed inpainting, outpainting, and style transfer. It also lands in a crowded month, with FireRedTTS3, Qwen-Audio-3.0, and dots.tts.edit all going after overlapping ground, so the open weights and MIT license are probably the most consequential part of the release for anyone building on top rather than picking a vendor.