vLLM-Omni adds Drama 3 speech serving

vLLM-Omni has merged first-class support for Fish Audio’s Drama 3, documented in the Drama 3 recipe. The backend serves the 35-billion-parameter text-to-speech model through the OpenAI-compatible /v1/audio/speech endpoint and streams 44.1 kHz mono audio. It supports voice-reference cloning, speaker tags, and bracketed acting directions while using vLLM’s batching and GPU execution stack.

One command, one endpoint

vllm serve fishaudio/drama-3 --omni --port 8091

Scripts carry the performance controls

Drama 3 is an open-weight mixture-of-experts model with about 3 billion parameters active for each token. Serving still requires storage for the full 35-billion-parameter weight set; routing lowers computation by activating only a subset. The model supports 83 languages, and tags from <|speaker:0|> through <|speaker:15|> allow as many as 16 characters in one request.

Inline directions shape delivery without a separate control format. For example, [sighs] Fine. If we miss it, it’s on you. [laughs] Well, maybe that’s fair. can change tone within a line. The recipe defines no fixed list of directions and reports that mid-sentence tags work more reliably than tags placed at the beginning.

Two decoding stages share a GPU

Stage 0 uses a Qwen3.5 mixture-of-experts backbone that combines Gated DeltaNet with standard full attention. Gated DeltaNet is a linear-attention variant that maintains recurrent state instead of extending a key-value cache for every token. The routed expert layers use block-FP8 quantization, which stores values in 8-bit floating-point blocks to reduce memory and compute costs.

For each audio frame, Stage 0 samples one main token. A 13-layer codebook tower, using its own cached attention history, predicts nine additional codes to produce a 10-codebook frame. Stage 1 passes those discrete codes to the Fish DAC decoder, which generates 44.1 kHz mono audio using the same codec as Fish Speech S2 Pro.

Both stages run on one accelerator. The default deployment configuration assigns a 0.75 GPU memory-utilization fraction to Stage 0 and 0.1 to the codec. Stage 0’s weights alone occupy 36.9 GiB, before accounting for caches, activations, and codec memory.

128 streams on an H200

On one H200, the optimized backend sustained 128 concurrent streams with a median time to first audio of 255 milliseconds and aggregate output of 222.2 seconds of audio per wall-clock second. The figures were averaged across five runs over a 1,088-request Seed-TTS-Eval English set. Voice references were pre-encoded, so the results exclude reference-encoding work.

Time to first audio measures the delay between request arrival and the first audio chunk. Real-time factor divides generation time by output duration, so values below 1 indicate faster-than-real-time synthesis. In the single-stream result, an RTF of 0.28 means one second of audio requires 0.28 seconds of generation time. Per-request latency rises as concurrent requests share the GPU, even when aggregate throughput improves.

Pronunciation scores and hard cases

Artificial Analysis’ Pronunciation Robustness test measures how accurately a model reads difficult words, numbers, and identifiers. Drama 3 scored 93.2%, ranking above Eleven v4 at 91.7% and Eleven v4 Turbo at 90.1% in the published comparison. Its weakest category was exact sequences such as codes, file paths, and email addresses, where it scored 86.3%.

Silence, nondeterminism, and GPU contention

• Some generations stop after the first spoken word or stage-direction event, then emit silence until max_tokens is reached. Lines beginning with a filler word or tag trigger this behavior more often.
• previous_takes, which supports continuation-based editing, is unavailable through the speech API.
• Long-text chunking is not exposed through the speech API, leaving applications to divide lengthy scripts themselves.
• Per-request seeds do not control the codebook heads, so identical seeds cannot produce fully deterministic audio across runs.
• The codec shares the GPU with model decoding and competes for execution time as concurrency increases.

A revenue-capped weight license

• Model weights: Fish Audio publishes the weights on Hugging Face under the Fish Audio Community License v1.0. The license permits free research and noncommercial use.
• Commercial use: Free commercial use applies when the licensee and its affiliates have less than US$500,000 in annual revenue. Commercial rights remain subject to the license’s other terms.
• Hosted API: Fish Audio lists a standard price of $50 per million characters and an announced promotional rate of $10 per million characters from October 13 through November 12.

Fewer serving components to maintain

Open-weight TTS deployments often require separate model inference, codec workers, request batching, and custom API code. The Drama 3 backend combines model and codec execution in one vLLM service, adds continuous batching, and uses CUDA graph replay to reduce repeated kernel-launch overhead. Voice agents, audiobook pipelines, game dialogue systems, and dubbing tools can use the same OpenAI-compatible interface while scaling to triple-digit concurrency on suitable hardware.