Tencent's AuK Handles 16 Audio Tasks From a Single Prompt
Tencent Hunyuan open-sources a 1.5B speech foundation model that handles TTS, editing, enhancement, and separation through plain-language instructions.
- Tencent Hunyuan open-sourced AuK, a 1.5B unified speech generation and editing foundation model.
- Handles 16 tasks including zero-shot TTS, instruct TTS, content editing, denoising, and source separation via natural-language prompts.
- Built on Qwen2.5-Omni-3B semantic encoder, a 50 Hz audio VAE, and a hybrid MMDiT plus single-stream DiT rectified-flow transformer.
- Trained on 3.03B instruction-audio pairs and 1.95M hours across five task families, with RLHF post-training.
- AuK-Flash distilled variant delivers 4-step inference and 4.5x speedup with CFG disabled.
- MIT-licensed, with weights on Hugging Face and ModelScope, plus Gradio, ComfyUI, and fine-tuning support.
Tencent’s AuK handles 16 audio tasks from one prompt
Tencent’s Hunyuan team, working with Shanghai Jiao Tong University and the Shanghai Innovation Institute, has released AuK on GitHub. The open-source audio foundation model uses a shared natural-language interface for voice cloning, speech generation, content editing, denoising, source separation, acoustic controls, and voice transformations. Its core generator has 1.5 billion parameters and depends on a separately downloaded Qwen2.5-Omni-3B model for semantic conditioning.
Conventional speech pipelines often combine text-to-speech, voice conversion, denoising, editing, and separation services. AuK moves those operations behind one message format, reducing the routing and integration code required to connect specialist systems. Developers provide an instruction, attach source or reference audio when needed, and receive a generated waveform.
Sixteen jobs, one message format
The AuK project site describes a chat-style API containing text and optional audio messages. A source clip provides material to transform, while a reference clip supplies characteristics such as speaker identity or style.
| Task family | Supported operations | Typical inputs |
|---|---|---|
| Generation | Zero-shot text-to-speech from a reference voice; speech generated from a written voice description | Target text plus a reference clip or voice description |
| Content editing | Replace, insert, or remove spoken words; rewrite lyrics while preserving melody and voice | Source audio plus an editing instruction |
| Acoustic control | Adjust pitch, speed, or volume using semitones, factors, or decibels | Source audio plus a numeric instruction |
| Paralinguistic editing | Change emotion or timbre; remove regional accents; add or remove breaths and laughter; convert between regular speech and whisper | Source audio plus the desired vocal change |
| Restoration and separation | Denoise and dereverberate audio; separate speakers by talking order; extract music vocals; isolate a speaker based on the words they say |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
