How Audio Event Detection Actually Works

Audio event detection is the problem of answering not just what sound is in this recording but when, and how many times. It is the difference between a label and a list of timestamps, and the gap between those two is where most of the engineering lives.

This is a walk through the whole chain, using a deliberately unglamorous example: finding burps and farts in a six-hour stream archive. The task is a joke and the pipeline is not — it is the same one used for gunshot detection, machine fault monitoring and baby monitors.

Step one: stop looking at the waveform

A raw waveform is amplitude over time, and almost nothing you want to detect is distinguishable by amplitude. A door slam and a cough can have identical envelopes. What separates sounds is how their energy is distributed across frequency, and how that distribution changes.

So the first move is always a time-frequency transform: chop the audio into short overlapping frames, and compute a spectrum for each. Stack those and you have a spectrogram — a picture, where one axis is time, the other is frequency, and brightness is energy.

Most pipelines then warp the frequency axis to a mel scale, which compresses the high end to roughly match how human hearing resolves pitch. Audio is also almost always downmixed to mono and resampled to 16 kHz first, because speech-and-noise models are trained there and nothing useful lives above 8 kHz for this purpose.

Step two: score every frame

The model takes a short window of that spectrogram and returns a score per class — for each hop forward in time, how confident it is that a burp is happening right now. Hop sizes are typically tens of milliseconds, so an hour of audio becomes tens of thousands of scores per class.

The output is not a set of events. It is a noisy curve that wobbles above and below plausibility thousands of times, and turning it into a list a person can use is the next three steps.

Step three: threshold, and mind the trade

Pick a value, and call anything above it an event. This single number controls the only trade-off that matters: a low threshold finds nearly everything and buries you in false positives, and a high one gives you a short, clean list that is missing things.

There is no correct setting, because the right answer depends on what the mistake costs. A detector feeding a review screen should err low — a human rejecting ten wrong guesses is cheap, and a missed moment is invisible and therefore unrecoverable. A detector triggering an alarm should err high.

Hysteresis: two thresholds, not one

A single threshold makes a mess at the boundary. A score hovering right at the line flickers above and below it, and you get one event chopped into nine.

The standard fix is borrowed from Schmitt triggers: use a high threshold to start an event and a lower one to continue it. A sound has to be clearly present to begin, then only plausibly present to carry on. One decisive onset, one clean run, no chatter.

Step four: clean up the geometry

What comes out of thresholding is still not a usable list. Three cheap passes do most of the remaining work, and they are pure interval arithmetic — no model involved:

• Smooth first. A median filter over the score curve removes single-frame spikes that no real sound could have produced.
• Drop the too-short. An event lasting one frame is noise. A minimum duration — tens of milliseconds, depending on the class — removes a large fraction of false positives at almost no cost to real detections.
• Merge the too-close. Two events separated by less than a short gap are one event with a dip in the middle. Merging them is what stops a single long sound being reported five times.
• Cap the too-long. Anything running for thirty seconds is not a burp, it is a noise floor shift or a microphone fault, and truncating or discarding it is more useful than reporting it.

The ordering matters and is easy to get wrong. Merging before dropping short events will fuse noise into real detections and inflate their durations; dropping before merging discards fragments that should have been joined. Smooth, threshold, drop, merge, cap.

Step five: accept that you need a human

No threshold produces a list that is both complete and clean. The honest design is to tune for recall, accept the false positives, and make rejecting them fast — a keyboard-driven review screen where each candidate takes two seconds to judge.

This is why precision matters less than it looks for this class of problem. A list of forty candidates containing all twelve real events is more useful than a list of twelve containing nine of them, because the first takes ninety seconds to clean and the second is permanently missing a quarter of the material.

Why detection sometimes finds nothing

When a pipeline like this returns zero events on material you know contains some, the cause is almost never the threshold. In rough order of likelihood:

1. The audio never arrived. A silent track, a muted VOD segment, or a video-only download with the audio stream missing.
2. The audio arrived degraded. Heavy compression destroys exactly the high-frequency transients these models key on.
3. The wrong channel. A detector fed only one side of a stereo recording where the microphone sits on the other.
4. Genuine absence, which is the explanation to reach for last rather than first.