Fish Audio Drama 3: Open-Source Text-to-Speech That Directs Performances, Not Just Reads Text

#From reading text to directing a scene

Most text-to-speech systems do one thing: you hand them a sentence, they read it back. That is fine for a navigation app, but it falls apart when you want an audiobook with a cast, a game cutscene, or a podcast with two hosts who actually sound like different people.

Fish Audio Drama 3 is built for that harder job. The team describes it as a model that turns text into directed performances. You can give it a script with several characters, clone voices from short reference clips, tell it in plain language how a line should land, and then say "do that again, but different" the way a director would. It is open source, and the weights are on Hugging Face.

#What is under the hood

Drama 3 is a mixture-of-experts model with 35B total parameters, of which only 3B are active for any given token. That keeps inference far cheaper than the headline number suggests. The text backbone mixes full attention with linear attention, a hybrid design that helps with long inputs.

On the audio side, a new codebook multi-token-prediction decoder generates 10 acoustic codebooks per frame in dedicated layers that attend over the entire sequence. The expert weights are stored in FP8 and everything else stays in bf16, which keeps memory use down.

The training recipe, as described in the technical report, has three stages of pre-training, then instruction fine-tuning, then large-batch reinforcement learning. The only reward in that final stage is a single learned model of human preference.

#What it can do

• Multi-character scenes. Speaker tags from <|speaker:0|> to <|speaker:15|> let one text carry up to 16 characters.
• Voice cloning. Clone a voice from one or more reference clips.
• Natural-language direction. Describe how a line should be delivered, and drop tags like [sighs] or [laughs] straight into the text.
• Multi-turn revision. Give a note on the previous take and the model revises it.
• Editing by infilling. Change a word or phrase in existing speech without re-recording the whole line.
• Long-form and streaming audio. Long text is chunked automatically and can be streamed chunk by chunk.
• 83 languages.

According to the paper, multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, and all of them live in one model.

#Hear it for yourself

The samples below all clone the same reference voice, so you can hear what changes from one request to the next.

#The reference voice

This is a 3.9-second clip from Common Voice, with the transcript "We asked over twenty different people, and they all said it was his." Every sample after this one tries to sound like this speaker.

Listen to the reference clip (en0.wav)

#Emotion and non-verbal events in the text

Tags are written at the spot where the event should happen. The request looks like this:

{
  "text": "[sighs] Fine. If we miss it, it's on you. [laughs] Well, maybe that's fair.",
  "references": [{
    "audio_path": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "text": "We asked over twenty different people, and they all said it was his."
  }]
}

Listen to the point-tags sample (A1_point_tags.wav)

#Editing a take like a sentence in a document

Here the model gets the earlier take as context and changes only what was asked. In this case "Wednesday" becomes "Thursday":

{
  "text": "The quarterly report is due on Thursday, so please finish it by noon.",
  "references": [{ "audio_path": "…", "text": "We asked over twenty different people, and they all said it was his." }],
  "previous_takes": ["The quarterly report is due on Wednesday, so please finish it by noon."]
}

Listen to the single-word edit (K1_edit_word_with_context.wav)

#Chaining a second edit

Edits stack. The next request changes "by noon" to "by five" and carries both earlier takes in the context:

{
  "text": "The quarterly report is due on Thursday, so please finish it by five.",
  "references": [{ "audio_path": "…", "text": "We asked over twenty different people, and they all said it was his." }],
  "previous_takes": [
    "The quarterly report is due on Wednesday, so please finish it by noon.",
    "The quarterly report is due on Thursday, so please finish it by noon."
  ]
}

Listen to the chained edit (K5_second_edit_chained.wav)

Want to try your own script? The live demo Space streams Drama 3 on ZeroGPU, and the Fish Audio Playground lets you direct performances in the browser.

#How fast is it?

Fish Audio measured streaming performance on NVIDIA H200 GPUs using Seed-TTS English prompts, with each request cloning a voice from one pre-encoded reference. Two terms matter here: time to first audio (TTFA) is how long a listener waits before sound starts, and real-time factor (RTF) is generation time divided by audio duration, so anything under 1.0 is faster than playback.

A few things stand out. A single stream gets its first audio in under 100 ms. At 16 concurrent streams, audio is still generated about 2.7 times faster than it plays. At 64 streams, mean RTF is 0.80.

Deployment choices matter too. At 256 concurrent streams, moving the vocoder from the engine's GPU to a second GPU raised throughput from 2,422 to 4,160 frames per second, about 72% more, and cut median TTFA from 3,403 ms to 2,146 ms. The paper also reports a single-stream decode rate of 271 frames per second on one H200, compared with 177 for Fish Audio S2, even though Drama 3 has more than seven times as many parameters.

#Run it yourself

Drama 3 is supported by two serving stacks from day one.

SGLang-Omni

git clone https://github.com/sgl-project/sglang-omni.git && cd sglang-omni
uv pip install --prerelease=allow -v -e ".[fishaudio]"

sgl-omni serve --model-path fishaudio/drama-3 --port 8000

vLLM-Omni

git clone https://github.com/vllm-project/vllm-omni.git && cd vllm-omni
uv pip install -e .

vllm serve fishaudio/drama-3 --omni --port 8091

The SGLang-Omni cookbook and the vLLM-Omni recipe cover installation, deployment, and API examples in more detail.

#The paper

The technical report, Fish Audio Drama 3 Technical Report, is available as arXiv 2610.18010 (PDF, Hugging Face paper page). Its authors are Dawei Han, Shidong Li, Shijia Liao, Jiahua Liu, Songting Liu, Qingzheng Wang, Yuxuan Wang, Xin Chen, Yifan Cheng, Tianyu Li, Gangin Park, Xuehao Yang, Chang Zeng, Haoyang Zhang, Ruoyi Zhang, and Xiangyu Zhang.

To cite it, a minimal BibTeX entry built from the paper page:

@misc{fishaudio2026drama3,
  title         = {Fish Audio Drama 3 Technical Report},
  author        = {Han, Dawei and Li, Shidong and Liao, Shijia and Liu, Jiahua and Liu, Songting and Wang, Qingzheng and Wang, Yuxuan and Chen, Xin and Cheng, Yifan and Li, Tianyu and Park, Gangin and Yang, Xuehao and Zeng, Chang and Zhang, Haoyang and Zhang, Ruoyi and Zhang, Xiangyu},
  year          = {2026},
  eprint        = {2610.18010},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2610.18010}
}

#License

Drama 3 is released under the Fish Audio Community License. Read the LICENSE file before using it commercially.

#Final thoughts

The interesting shift in Drama 3 is not just size or speed. It treats speech generation as an iterative, directable process: cast the scene, record a take, give a note, fix one word, and go again. Having that in an open model, with serving support in both SGLang-Omni and vLLM-Omni, makes it easy to experiment with.

#Resources

• Model weights: huggingface.co/fishaudio/drama-3
• Live demo: huggingface.co/spaces/fishaudio/Drama-3
• Technical report: arxiv.org/abs/2610.18010
• Fish Speech on GitHub: github.com/fishaudio/fish-speech
• Fish Audio Playground: fish.audio
• SGLang-Omni: sgl-project.github.io/sglang-omni
• vLLM-Omni recipe: recipes.vllm.ai/fishaudio/drama-3