Fish Audio's Drama 3 Lets Developers Direct AI Voices Like a Film Script
Fish Audio's new 35B mixture-of-experts text-to-speech model ships open weights, with plain-language direction, mid-sentence voice shifts, and word-level audio inpainting.
- Fish Audio released Drama 3, an open-weight directable TTS model with natural-language stage directions.
- 35B mixture-of-experts architecture with 3B active parameters per token, published on Hugging Face.
- Audio inpainting lets you regenerate a single word while keeping surrounding voice, mood, and rhythm intact.
- Multi-speaker scenes render in one pass; supports mid-sentence voice and language shifts across EN, ZH, JA, KO.
- Available via API as
drama-3-previewat $15 per 1M UTF-8 bytes, plus partner platforms like ComfyUI and Runway. - Day-zero inference support from SGLang; backed by a $52M seed round and 15M+ claimed users.
Fish Audio has released Drama 3, a text-to-speech model designed to generate directed scenes from annotated scripts. Developers can place natural-language performance cues alongside dialogue, including instructions that change delivery within a sentence. The 35-billion-parameter mixture-of-experts model is available as open weights on Hugging Face and through the Fish Audio API under the identifier drama-3-preview.
The launch follows a $52 million seed round and a company-reported user base of more than 15 million. Drama 3 is Fish Audio’s first flagship directable model released in the open-weight tier, giving teams the option to use a hosted endpoint or run the checkpoint on their own infrastructure.
Direction lives in the script
Drama 3 reads performance cues embedded in English, Chinese, Japanese, and other supported languages. Square-bracket instructions can set delivery from their position in the script or insert events such as a sigh. Paired tags such as <whisper> limit an effect to the enclosed words.
- Mid-line direction: A sentence can begin calmly and become panicked without generating separate takes.
- Multi-speaker scenes: Several characters can appear in one generation while retaining distinct voices.
- Audio inpainting: A selected word or segment can be regenerated while the surrounding audio remains in place.
- Long-form multilingual delivery: The model can preserve a voice across extended scripts, switch languages within a scene, and accept phoneme-level pronunciation controls.
I'm fine. Really. <whisper>Go on without me.</whisper>
Alright, [sigh] let's try again.
[warm and relaxed, like talking to an old friend] It's been a while.Fish Audio also provides Auto Tag, which proposes an initial set of performance directions for unannotated dialogue.
Repairs stay inside the take
Audio inpainting conditions a replacement on the sound immediately before and after the selected region. That context helps the new segment match the original speaker, mood, timing, and rhythm, reducing the need to regenerate an entire line after a small script change.
“When you change a single word, Drama 3 listens to the audio on both sides of it and makes the new word fit. Same voice, same mood, same rhythm. It sounds like a small thing, but it was one of the hardest parts of the model to get right,” said Shijia Liao, Fish Audio co-founder and chief scientist.
For dialogue-heavy games, audiobooks, localization pipelines, and conversational agents, local repair could reduce repeated generation and manual audio splicing. The preview also supports voice changes within a sentence and multi-character rendering in a single pass.
Sparse compute, full-scene context
Drama 3 contains 35 billion parameters and activates roughly 3 billion for each token. Its mixture-of-experts router selects a subset of specialized model components during inference, reducing compute compared with activating every parameter for every token. The 3-billion figure describes per-token computation; memory planning must still account for the complete checkpoint, precision, runtime cache, and context length.
Fish Audio traces the multi-speaker system to its S2 model family. S2 can process reference audio containing several voices and assign each one a speaker token. Drama 3 extends that approach to generate a scene as one audio stream, helping characters respond with consistent pacing instead of requiring independently generated clips.
For long scripts, Fish Audio says the model maintains timbre and prosody across a continuous generation. That capability targets audiobooks and multi-minute cinematics, where chunked requests can introduce audible changes in voice or pacing. The company has not yet published a maximum practical context length or independent long-form benchmarks.
Three routes into production
- Hosted API: Send the annotated script through REST or WebSocket streaming with
drama-3-previewas the model header. Fish Audio provides Python and TypeScript SDKs. - Self-hosted weights: Download the checkpoint from Hugging Face and use the fish-speech tools. The runtime detects the model from
config.json, and Drama 3 shares its command-line interface and API server with S2-Pro. Fish Audio credits SGLang and Radix Ark with launch-day inference support. - Partner platforms: Launch integrations include ComfyUI, Runway, Pika Labs, OpenRouter, Vercel, Creatify, OpenArt, and Nous Research’s Hermes agent.
Minimal hosted request
curl https://api.fish.audio/v1/tts \
-H "Authorization: Bearer $FISH_API_KEY" \
-H "model: drama-3-preview" \
-H "Content-Type: application/json" \
-d '{
"text": "I'm fine. Really. <whisper>Go on without me.</whisper>",
"reference_id": "f3a2b90078d54a65af4b95f63bc7798e"
}' --output speech.mp3Hosted generation costs $15 per 1 million UTF-8 bytes. Because billing counts encoded bytes, scripts containing non-ASCII characters can consume the allowance faster than the same number of basic Latin characters.
Best fits and preview limits
| Workload | Relevant capability |
|---|---|
| Game dialogue | Local line repairs and multi-character scene generation |
| Audiobooks | Long-form voice and pacing consistency |
| Conversational agents | Emotional changes within a response |
| Localization | Cross-language voice consistency and target-language directions |
The preview carries several constraints that affect production planning:
- Some launch channels distribute access keys only after a user comments to request one.
- Fish Audio has not published complete quality metrics, a full language list, production GPU requirements, or quantization guidance.
- No public migration guide explains the changes required for applications built around S2.1.
- Results can vary between generations, so applications may need candidate selection or retry logic.
- Open-weight refers to checkpoint access. Production teams still need to review the model license, deployment terms, and voice-consent requirements.
A wider control surface for open TTS
Expressive TTS workflows often combine reference recordings, preset style controls, SSML, repeated generation, and manual editing. Drama 3 moves more of that work into the script while exposing local repair and multi-speaker generation through hosted and self-managed deployments. Its practical value will depend on instruction accuracy, edit quality, latency, memory use, and long-form consistency when tested against representative production scripts.
Comments
