Interactive TTS Teaches Voice AI to Whisper, Shout, and Grieve on Cue
A new research paper proposes decoupling style decisions from waveform generation, using explicit instructions and preference optimization to keep voices consistent across turns.
- New research paper proposes Interactive TTS, decoupling style planning from speech generation.
- Explicit style instructions replace implicit end-to-end mapping, fixing timbre drift across turns.
- Two training tricks: Iterative Rejection Sampling Fine-Tuning and Context-Aware DPO.
- Reports state-of-the-art results on VStyle and SpeechParaling-Bench benchmarks.
- Live demo site shows Chinese responses across whispered, urgent, and comforting scenarios.
- No code or checkpoint released; built on Qwen3.5-Omni Thinker-Talker architecture.
Interactive TTS gives voice models a written delivery plan
Wenjie Tian and collaborators propose Interactive TTS, a framework that converts conversational context into a written speaking-style instruction before synthesizing audio. The design aims to vary pace, volume, and emotion while preserving the speaker’s identity and following user cues. The research paper reports stronger results than previous leading systems on two expressive-speech benchmarks.
The team’s project demos present Chinese-language responses to situations including a kitchen fire, a sleeping baby in the next room, and a user grieving a pet. These scenarios test whether one reference voice can adapt its delivery to conversational and visual context.
Why a voice drifts with the dialogue
Context-aware text-to-speech systems commonly feed dialogue history into an end-to-end model and leave it to infer whether the response should be whispered, shouted, slowed down, or softened. Style, linguistic content, and timbre, the acoustic qualities that identify a speaker, can become entangled inside the model. That makes style decisions difficult to supervise and can cause inconsistent voices or missed instructions across turns.
Interactive TTS assigns style selection and audio rendering to separate components. One module determines how the response should sound, while another generates the waveform from the reply text, style instruction, and voice reference.
A planner sits in front of the synthesizer
| Component | Inputs | Output | Purpose |
|---|---|---|---|
| Context-to-Instruction planner | Dialogue history and available multimodal cues, including video in some demos | A short style instruction | Select an appropriate delivery for the current situation |
| TTS generator | Reply text, style instruction, and fixed voice reference |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
