← All notes
Published
Reading
14 MIN READ
Field notes //

Speech Event Detection: Beyond Words Towards Richer Speech Transcription

Modern ASR systems like Whisper are effective at transcribing speech, but they often miss important non-verbal vocalizations and environmental sounds such as laughter, breathing, sighs, coughs, music, and sirens.Speech Event Detector addresses this gap by detecting these events, assigning precise timestamps, and integrating them with existing speech transcripts to create richer, more informative audio representations.

1. The Problem: ASR Captures Words, Not the Entire Audio

Consider a simple conversation:

"I think we should go tomorrow. [sigh] Actually, I'm not sure."

A conventional ASR system might produce:

"I think we should go tomorrow. Actually, I'm not sure."

The words are captured, but the sigh disappears.

Now consider another recording:

  • A person is speaking.
  • A dog barks in the background.
  • A vehicle horn interrupts the conversation.
  • The speaker laughs before completing a sentence.

A conventional transcript may capture the spoken words while omitting these events entirely.

This creates a gap between the original audio and its textual representation.

There are several applications where this missing information matters:

  • Conversation analysis: Laughter, sighs, pauses, and other vocalizations can provide useful context for studying conversational interactions.
  • Expressive speech research: Non-verbal vocalizations are important when building datasets for expressive TTS and speech generation.
  • Audio understanding: Environmental sounds can provide contextual information about the recording scene.
  • Multimodal AI: Combining linguistic content with sound events creates a richer representation of audio and video.
  • Dataset curation: Identifying non-verbal events can help filter, annotate, and organize large speech datasets.

The objective is not to replace ASR. Instead, it is to complement it with another layer of information.

2. Introducing Speech Event Detector

Speech Event Detector is a compact convolutional–Transformer model that identifies non-verbal vocalizations and background sounds in audio recordings.

It predicts the probability of different sound events at 40 ms intervals and converts these predictions into timestamped event segments.

The model supports 54 event classes, divided into two major categories.

Speaker vocalizations (20 classes)

These represent sounds produced by the speaker beyond ordinary linguistic speech.

Examples include:

  • Breathing
  • Sighing
  • Sniffing
  • Laughing
  • Coughing
  • Throat clearing
  • Crying
  • Gasping
  • Yawning
  • Snoring
  • Sneezing
  • Nose blowing
  • Lip smacking
  • Groaning
  • Screaming
  • Whispering
  • Shouting
  • Chewing
  • Burping
  • Hiccupping

Background sounds (34 classes)

These represent sounds originating from the surrounding environment.

Examples include:

  • Vehicle sounds
  • Horns
  • Sirens
  • Birds
  • Insects
  • Dog barking
  • Animal sounds
  • Child voices
  • Baby crying
  • Music
  • Television and speakers
  • Machines and fans
  • Phone ringing and vibration
  • Beeps and bells
  • Whistles and buzzers
  • Applause and cheering
  • Crowds
  • Footsteps
  • Wind, rain, and thunder
  • Water
  • Knocking and doors
  • Dishes and glass breaking
  • Gunshots and explosions
  • Construction sounds

The distinction between speaker vocalizations and environmental sounds is important because they have different temporal characteristics and different requirements for transcript representation.

3. From Audio to Timestamped Events

The central idea behind the system is to transform an audio recording into a sequence of timestamped sound events.

Instead of producing a single classification for an entire recording, the detector generates frame-level probabilities.

For every 40 ms interval, the model estimates the probability of each of the 54 event classes.

The output can be represented as:

(event_class, start_time, end_time, confidence)

For example:

[
  {
    "label": "sigh",
    "start": 1.24,
    "end": 2.08,
    "prob": 0.91
  },
  {
    "label": "vehicle",
    "start": 4.12,
    "end": 6.85,
    "prob": 0.87
  },
  {
    "label": "laugh",
    "start": 8.30,
    "end": 9.15,
    "prob": 0.94
  }
]

Illustrative output showing the detector's event schema, not measured predictions from a particular recording.

This temporal representation is important because knowing that a recording contains laughter is not enough. We also need to know when the laughter occurred and how it relates to the surrounding speech.

The system uses overlapping audio windows, temporal smoothing, class-specific thresholds, and event-merging rules to convert frame-level predictions into meaningful segments.

Running inference with Python

The released model can be used directly from Python.

First, install the dependencies and download the pretrained model:

pip install torch torchaudio numpy scipy safetensors huggingface_hub

# Optional: install Whisper for automatic transcription
pip install openai-whisper

# Download the model
hf download theboringai-work/surround-sound \
    --local-dir surround-sound

From the downloaded project directory, load the detector and run inference:

from sed import Detector

# Load the pretrained detector
detector = Detector.from_pretrained(
    "theboringai-work/surround-sound",
    profile="sensitive"
)

# Run event detection
events = detector.detect("recording.mp3")

# Inspect detected events
for event in events:
    print(
        event["label"],
        event["start"],
        event["end"],
        event["prob"]
    )

The detector supports two operating profiles:

  • sensitive: Prioritizes event recall, allowing more candidate detections.
  • balanced: Uses a more conservative threshold configuration.

The choice depends on the application. For dataset annotation and research exploration, a more sensitive profile may be useful. For automatically generated transcripts, a balanced profile may produce a more conservative result.

Example output

A detection result contains the event label, start time, end time, and confidence probability.

[
    {
        "label": "cough",
        "start": 0.68,
        "end": 2.00,
        "prob": 0.92
    },
    {
        "label": "breath",
        "start": 7.72,
        "end": 8.72,
        "prob": 0.94
    }
]

These timestamps make the predictions useful for downstream applications that require temporal alignment rather than recording-level classification.

4. Model Architecture

We designed the detector as a relatively small convolutional–Transformer network rather than relying on a large general-purpose audio model.

The model contains approximately 5.8 million parameters.

Its architecture combines two complementary components.

Convolutional feature extraction

The input audio is resampled to 16 kHz mono and converted into a 64-band log-mel spectrogram.

Five convolutional blocks extract local time-frequency patterns from the spectrogram.

These layers learn acoustic characteristics that help distinguish events such as coughing, laughter, breathing, horns, and other environmental sounds.

Transformer-based temporal modelling

The convolutional features are projected into a 256-dimensional representation and passed through six Transformer encoder layers.

Each layer uses eight attention heads.

The Transformer allows the model to incorporate temporal context across the complete eight-second input window.

This is particularly useful because sound events cannot always be identified reliably from a very short acoustic fragment. Their surrounding context can help distinguish similar sounds.

Finally, a classification head with sigmoid activations produces probabilities for all 54 event classes at 40 ms resolution.

Architecture overview

Input Audio
16 kHz Mono
     |
     v
Log-Mel Spectrogram
64 Mel Bands
     |
     v
5 Convolutional Blocks
Local Acoustic Features
     |
     v
Feature Projection
768 -> 256 Dimensions
     |
     v
6 Transformer Encoder Layers
8 Attention Heads
     |
     v
Classification Head
54 Sigmoid Outputs
     |
     v
Frame-level Probabilities
40 ms Resolution
     |
     v
Temporal Post-processing
     |
     v
Timestamped Sound Events

The resulting architecture combines local acoustic feature extraction with broader temporal context while keeping the model relatively compact.

5. The Data Challenge: Bringing Together Incompatible Datasets

One of the most challenging parts of this work was not designing the neural network. It was preparing the training data.

Public datasets containing non-verbal vocalizations and environmental sounds use different annotation formats.

Some provide timestamps for individual events. Others contain free-text labels, ontology-based annotations, isolated sound recordings, or event symbols embedded directly inside transcripts.

These formats cannot be used together without additional processing.

We addressed this by developing a rule-based label-unification pipeline that maps seven public data sources into a common event taxonomy.

The sources include:

  1. NonVerbalSpeech-38K: A collection of non-verbal speech clips, predominantly Mandarin, with detected event regions.
  2. Vaani Noise Event Dataset: Environmental and vocal sound annotations across Indian languages, with free-text event labels and timestamps.
  3. NonverbalTTS: An English corpus containing non-verbal vocalizations embedded in transcripts.
  4. AudioSet-Strong: Temporally annotated environmental sound events.
  5. OpenSLR 99: Isolated recordings of non-verbal vocalizations.
  6. DCASE Domestic and Office Noise Corpus (DNC): Environmental recordings used for background sound augmentation.
  7. DEMAND: Environmental noise recordings from different acoustic environments.

Each source contributes different information.

For example, NonVerbalSpeech-38K provides a substantial collection of vocalization recordings, while Vaani contributes diverse real-world noise events from Indic speech environments.

AudioSet-Strong provides temporally annotated environmental sounds, and NonverbalTTS provides vocalization labels embedded in speech transcripts.

We mapped the available annotations into a unified taxonomy using explicit rules rather than assuming that all datasets used identical label definitions.

Example: Label unification

Consider different datasets that use different names for similar acoustic events.

A mapping layer converts these annotations into the detector's common taxonomy.

# Illustrative representation of the label-mapping concept

LABEL_MAP = {
    "laughing": "laugh",
    "laughter": "laugh",
    "chuckling": "laugh",

    "breathing": "breath",
    "inhalation": "breath",

    "sighing": "sigh",
    "coughing": "cough",

    "car_horn": "horn",
    "vehicle_horn": "horn",
    "siren_sound": "siren",
}

def normalize_label(source_label):
    label = source_label.lower().strip()
    return LABEL_MAP.get(label, label)

This is a simplified illustration of the mapping strategy, not the complete mapping implementation used in training.

The unified annotations allowed us to combine different sources into a common training pipeline.

6. Recovering Missing Timestamps with CTC Forced Alignment

An interesting challenge appeared when working with NonverbalTTS.

The dataset identifies non-verbal sounds inside transcripts using symbols, but these annotations do not directly provide precise audio timestamps.

For example:

"The speaker was talking [laugh] and then continued."

We know where the laugh occurs relative to the words, but not its exact start and end times in the audio.

To address this, we used CTC-based forced alignment.

The process involved:

  1. Aligning transcript words to the audio using the MMS forced-alignment model.
  2. Identifying the pause between the words surrounding each non-verbal event.
  3. Searching for the event within the corresponding acoustic region.
  4. Estimating its boundaries using frame-level energy and duration constraints.

Using this procedure, we located 6,360 out of 8,620 annotated sounds, approximately 74%.

The recovered timestamps were then used to create additional event crops and annotated training windows.

An important qualification is that these recovered event boundaries were not manually verified. They inherit potential errors from transcript alignment and the event-placement procedure.

Conceptual example of forced alignment

A forced aligner maps known transcript tokens to positions in an audio recording.

# Conceptual illustration of transcript alignment

transcript = [
    "I",
    "think",
    "[laugh]",
    "we",
    "should",
    "go"
]

# Word-level alignment from an audio aligner
aligned_words = [
    ("I", 0.10, 0.24),
    ("think", 0.25, 0.62),
    ("we", 1.15, 1.34),
    ("should", 1.35, 1.70),
    ("go", 1.71, 1.95),
]

# The non-verbal token occurs between
# "think" and "we".
left_word = aligned_words[1]
right_word = aligned_words[2]

search_region = (
    left_word[2],
    right_word[1]
)

print("Search region:", search_region)

This illustrates how word boundaries can constrain the region in which a transcript-embedded event is searched for. The actual training pipeline additionally uses acoustic information and duration constraints to estimate event boundaries.

Nevertheless, this provided a practical way to extract additional temporal supervision from data that did not originally contain explicit event timestamps.

7. Combining Real Recordings with Synthetic Audio Mixtures

Training a sound event detector requires temporally annotated examples.

However, strongly annotated audio is limited, particularly for events that occur naturally during speech.

To increase the diversity of training examples, we combined real annotated recordings with synthetic mixtures.

The synthetic pipeline combines:

  • Speech recordings used as background beds.
  • Isolated non-verbal vocalizations.
  • Environmental sound crops.
  • Real background noise.

Because the event positions are known when sounds are mixed, synthetic mixtures provide accurate event-placement supervision by construction.

Conceptual example: Creating a synthetic mixture

The following simplified example illustrates how an isolated sound event can be mixed into a speech recording while retaining its timestamp.

import numpy as np

def mix_event(speech, event, start_sample, gain=0.7):
    """
    Add an event crop to a speech waveform.

    Both waveforms must use the same sample rate.
    """

    mixed = speech.copy()

    end_sample = min(
        start_sample + len(event),
        len(mixed)
    )

    event_length = end_sample - start_sample

    mixed[start_sample:end_sample] += (
        gain * event[:event_length]
    )

    return mixed

# Example placement at 3 seconds for 16 kHz audio
sample_rate = 16000
start_sample = int(3.0 * sample_rate)

# mixed_audio = mix_event(
#     speech_audio,
#     laugh_audio,
#     start_sample
# )

In this example, the event begins at exactly three seconds.

The actual training pipeline includes additional steps for waveform preparation, mixing levels, augmentation, and target generation.

We also applied augmentation to both real and synthetic recordings.

The resulting training pipeline used approximately 168 hours of real annotated recordings, together with synthetic mixtures and augmented examples.

The model was trained using eight-second audio windows with frame-level supervision.

This combination allowed us to use heterogeneous public datasets while increasing the diversity of acoustic conditions seen during training.

8. Integrating Sound Events with ASR Transcripts

Detecting sound events independently is useful, but their real value increases when they are combined with speech transcripts.

The system includes a transcript-merging procedure that combines detected events with word-level timestamps from an ASR system.

Our paper demonstrates this integration using Whisper.

The important distinction is that the detector is not designed to replace the speech recognizer.

Instead, it operates as an additional component that enriches an existing transcription.

Running the complete pipeline

The repository provides a command-line interface that can run event detection together with Whisper transcription.

# Run event detection and Whisper transcription
python inference.py recording.mp3

# Run event detection without ASR
python inference.py recording.mp3 --asr none

# Process an entire directory
python inference.py recordings/ --asr none

# Export machine-readable event results
python inference.py recordings/ \
    --asr none \
    --json events.jsonl

This makes it possible to use the detector independently or as part of a transcription pipeline.

Using the detector independently

For applications that already have their own ASR system, the detector can be used without running another transcription model.

from sed import Detector

detector = Detector.from_pretrained(
    "theboringai-work/surround-sound",
    profile="balanced"
)

events = detector.detect("recording.wav")

for event in events:
    print(
        f'{event["label"]}: '
        f'{event["start"]:.2f}s - '
        f'{event["end"]:.2f}s'
    )

The resulting events can then be passed to an existing transcription or annotation system.

Accessing frame-level probabilities

Sometimes event segments alone are not sufficient.

For research applications, it can be useful to inspect the model's underlying probability predictions.

The detector also exposes frame-level probabilities:

from sed import Detector

detector = Detector.from_pretrained(
    "theboringai-work/surround-sound",
    profile="sensitive"
)

# audio_16k should be a 16 kHz mono waveform
# represented as a NumPy array.

times, probs = detector.frame_probs(audio_16k)

print("Time steps:", len(times))
print("Probability matrix:", probs.shape)

The probability matrix contains the model's predictions across the 54 event classes at 40 ms resolution.

This is useful for:

  • Inspecting event confidence over time.
  • Developing custom thresholding strategies.
  • Studying overlapping events.
  • Visualizing event activation patterns.
  • Building custom temporal post-processing.

Merging with an existing ASR transcript

The transcript-merging functionality is not restricted to Whisper.

Any ASR system that provides compatible word-level timestamps can be integrated.

For example:

from sed import Detector, rich_transcript, clean

detector = Detector.from_pretrained(
    "theboringai-work/surround-sound",
    profile="sensitive"
)

# Detect non-verbal and background events
events = detector.detect("recording.mp3")

# Example word-level ASR output:
# (word, start_time, end_time)

words = [
    ("I", 0.10, 0.24),
    ("think", 0.25, 0.62),
    ("we", 1.15, 1.34),
    ("should", 1.35, 1.70),
    ("go", 1.71, 1.95),
]

# Clean and merge detected events with ASR words
transcript = rich_transcript(
    words,
    clean(words, events)
)

print(transcript)

The words variable represents an example of word-level output from an ASR system. In a production pipeline, it would be populated with the actual recognizer's timestamped predictions.

Example of enriched transcription

A conventional transcript might look like:

I think we should go tomorrow.
Actually, I'm not sure.

An enriched transcript can represent the additional information:

I think we should go tomorrow. [sigh]
Actually, I'm not sure.

Background events can also be represented using opening and closing tags:

I was walking down the <vehicle> road </vehicle>
when I heard someone laughing.

The merging procedure preserves the recognizer's word order and inserts event annotations according to their timestamps.

It also handles overlapping events, merges nearby detections of the same class, and removes certain events already represented by ASR filler or laughter tokens.

Although Whisper is used in the paper's implementation, the timestamped event detection component is designed to integrate with transcription systems that provide compatible word-level timestamps.

This makes the approach useful beyond a single ASR architecture.

9. Evaluation and Results

We evaluated the detector on held-out real recordings from four datasets.

The evaluation considered 48 event classes with at least ten annotated events in the held-out recordings, covering a total of 9,367 events.

Rather than relying on a single classification threshold, we introduced two operating profiles.

Sensitive profile

The sensitive profile prioritizes finding more events, even if this results in additional false positives.

It is more suitable for applications where missing an event is costly and a human can review the resulting annotations.

Results:

  • Macro event recall: 0.491
  • Macro event precision: 0.302
  • Macro event F1: 0.322
  • Event-weighted recall: 67.2%

Balanced profile

The balanced profile attempts to maintain a better trade-off between precision and recall.

It is more suitable for automated transcription pipelines where excessive false annotations can reduce usability.

Results:

  • Macro event recall: 0.369
  • Macro event precision: 0.361
  • Macro event F1: 0.329
  • Event-weighted recall: 53.3%

The sensitive profile identifies approximately two-thirds of all annotated events, while the balanced profile offers a more conservative operating point.

Some event categories performed considerably better than others.

For example, music, sighs, horns, laughter, sniffing, birds, and breathing showed comparatively strong event-level results under the sensitive profile.

Other categories, including yawn, fan, door, rain, chewing, buzzer, and television speaker, remained challenging.

These differences highlight an important characteristic of real-world sound event detection: performance depends heavily on event frequency, annotation quality, acoustic similarity, and recording conditions.

Computational efficiency

The detector is relatively lightweight.

On the reported hardware:

  • Eight CPU threads: approximately 39× real-time.
  • NVIDIA T4 GPU: approximately 380× real-time.

This means that 178 seconds of audio took approximately 4.6 seconds on eight CPU threads, excluding decoding and model loading.

These measurements refer to the event detector itself. Additional ASR processing introduces its own computational cost.

10. Applications: Where Can Rich Speech Transcription Be Useful?

The motivation behind this work extends beyond adding labels to transcripts.

A timestamped representation of linguistic and non-linguistic audio events can serve as a foundational component for several research and engineering applications.

Conversation and interaction analysis

A conversation is not simply a sequence of words.

Laughter, sighs, breaths, pauses, and other vocal events can provide useful signals for understanding how an interaction unfolds.

A richer transcript can help researchers investigate how these events relate temporally to spoken language.

Expressive speech and TTS datasets

Building expressive speech systems requires more than collecting clean text-to-speech pairs.

Non-verbal vocalizations such as laughter, sighing, gasping, and breathing are important components of natural speech.

Timestamped event annotations can support dataset discovery, filtering, and the preparation of more expressive speech corpora.

Audio and video understanding

When audio is associated with video, speech and sound events can be aligned with visual information.

For example, an audio event such as applause can be aligned with a scene transition, or a siren can provide additional information about an outdoor recording.

This creates opportunities for richer multimodal representations.

Long-form audio processing

Long recordings often contain several different acoustic events that are not represented in ordinary transcripts.

A timestamped event layer can make it easier to locate particular moments, identify changes in acoustic conditions, and organize long-form audio for downstream analysis.

Speech dataset quality and curation

Sound events can also help identify useful or problematic regions in speech datasets.

For example, researchers may want to locate recordings containing background music, overlapping vocalizations, coughing, or other acoustic conditions before selecting data for training.

These applications are potential directions enabled by richer audio annotations, rather than claims that the detector independently solves all these tasks.

11. Limitations and Future Directions

There are several limitations that are important to acknowledge.

First, the evaluation is not based on a completely independent final test set. Checkpoint selection and threshold tuning used validation windows from the same held-out recordings as the event-level evaluation. Consequently, the reported absolute performance may be optimistic.

Second, the model was trained using heterogeneous datasets with different recording conditions and annotation practices. Its performance on domains such as call-centre recordings, meetings, far-field microphones, and other specialized environments remains unknown.

Third, some event classes have limited training examples or weak detection performance.

Finally, the current architecture does not explore pretrained audio encoders such as BEATs or WavLM, alternative window lengths, or extensive component ablations.

These are useful directions for future work.

Some areas we would like to explore include:

  • Improving detection of short-duration vocalizations.
  • Better handling of overlapping speech and non-verbal events.
  • More reliable event boundaries and temporal localization.
  • Expanding coverage across languages and recording environments.
  • Evaluating the detector on independent, carefully annotated target-domain datasets.
  • Exploring integration with larger audio-language and multimodal models.
  • Studying how non-verbal events and environmental sounds contribute to conversational context.

The longer-term goal is to move towards audio representations that preserve more than linguistic information.

12. Try Speech Event Detector

We have released the implementation, pretrained weights, and an interactive demonstration to make the work accessible to researchers and developers.

You can explore the project through the following resources:

The released model weights are distributed under CC BY-NC-SA 4.0 terms, reflecting licensing restrictions inherited from some of the training sources. Please review the repository and model documentation before using the model.

Citation

If you use this work in your research, please cite:

@misc{bar2026speecheventdetector,
  title = {
    Speech Event Detector: Timestamped Detection of
    Non-Verbal Vocalizations and Background Sounds
    for Rich Speech Transcription
  },
  author = {
    Bar, Ayush Kumar and Ghosh, Promila
  },
  year = {2026},
  eprint = {2610.14232},
  archivePrefix = {arXiv},
  primaryClass = {cs.SD},
  url = {https://arxiv.org/abs/2610.14232}
}

Conclusion

Speech recognition has made significant progress in understanding what people say. However, understanding an audio recording requires more than recognizing its words.

Breathing, laughter, sighs, coughs, environmental sounds, and other acoustic events are part of the original recording and can carry information that conventional transcripts do not preserve.

With Speech Event Detector, we explore a practical approach to adding this missing layer to speech transcription.

By combining a compact convolutional–Transformer architecture, heterogeneous public datasets, timestamp recovery, synthetic audio mixtures, and ASR transcript integration, we take a step towards richer and more informative audio representations.

The goal is simple: move beyond transcripts that capture only what was said, towards representations that capture more of what actually happened in the audio.

References

  1. Bar, A. K., and Ghosh, P. (2026). Speech Event Detector: Timestamped Detection of Non-Verbal Vocalizations and Background Sounds for Rich Speech Transcription.
    https://arxiv.org/abs/2610.14232

  2. Whisper: Robust Speech Recognition via Large-Scale Weak Supervision.
    https://arxiv.org/abs/2212.04356

  3. BEATs: Audio Pre-Training with Acoustic Tokenizers.
    https://arxiv.org/abs/2212.09058

  4. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing.
    https://arxiv.org/abs/2110.13900

  5. AudioSet: An Ontology and Human-Labeled Dataset for Audio Events.
    https://research.google/pubs/audioset-an-ontology-and-human-labeled-dataset-for-audio-events/

  6. NonverbalTTS.
    https://huggingface.co/datasets/deepvk/NonverbalTTS

  7. Vaani Noise Event Dataset – ARTPARK / IISc.
    https://huggingface.co/datasets/ARTPARK-IISc/Vaani-Noise-Event-Dataset

  8. NonVerbalSpeech-38K.
    https://huggingface.co/datasets/nonverbalspeech/nonverbalspeech38k

  9. AudioSet-Strong.
    https://huggingface.co/datasets/enyoukai/AudioSet-Strong

  10. OpenSLR 99.
    https://www.openslr.org/99/

This article describes the research presented in the preprint. Experimental results and limitations are based on the evaluation documented in the paper.