Field
notes
Write-ups from the work: fine-tuning ASR and TTS for Indic languages, serving speech models on vLLM and Triton, building voice agents, and what broke along the way.
-
01 5 min read
Fish Audio Drama 3: Open-Source Text-to-Speech That Directs Performances, Not Just Reads Text
Fish Audio Drama 3 is an open-source, 35B-parameter mixture-of-experts speech model (3B parameters active per token) built for multi-character scenes and long-form audio. One model handles voice cloning, natural-language direction, multi-turn revision, editing by infilling, and streaming across 83 languages. On a single H200 GPU, a stream's first audio arrives in under 100 ms.
-
02 14 min read
Speech Event Detection: Beyond Words Towards Richer Speech Transcription
Modern ASR systems like Whisper are effective at transcribing speech, but they often miss important non-verbal vocalizations and environmental sounds such as laughter, breathing, sighs, coughs, music, and sirens.Speech Event Detector addresses this gap by detecting these events, assigning precise timestamps, and integrating them with existing speech transcripts to create richer, more informative audio representations.
-
03 3 min read
OpenAI Whisper Inference Guide
OpenAI Whisper is an open source automatic speech recognition system developed by OpenAI. It is built on a Transformer encoder decoder architecture and trained on a large scale multilingual dataset collected from the web. Whisper is capable of transcribing speech in multiple languages, identifying the spoken language, and translating speech directly into English. Inference with Whisper is straightforward. After installing the package, a pretrained model can be loaded with a single function call. The model automatically handles audio preprocessing, including resampling to 16 kHz and conversion to log Mel spectrogram features. Once the audio is processed, the decoder generates text autoregressively, producing a transcription along with metadata such as detected language and segment timestamps. Whisper supports multiple model sizes, allowing users to balance accuracy and computational cost. Smaller models are suitable for low latency or edge environments, while larger models offer improved robustness to accents and background noise. GPU acceleration through PyTorch significantly improves inference speed, especially for medium and large checkpoints.
- No posts with that tag yet.