Spectrograms

Time and frequency at once

A waveform shows when energy happens but not which frequencies. A spectrum shows frequencies but averages over the whole frame. The short-time Fourier transform (STFT) gives both by computing a spectrum for every frame and stacking them. Time runs along one axis, frequency along the other, and brightness is magnitude (how much energy a frequency holds). That picture is the spectrogram. It is the most useful single view of speech ever invented. Once you can read one, debugging an audio pipeline becomes visual.

The transform is the same one from the previous chapter, applied many times. A 25 ms frame with a 10 ms hop produces one spectrum every 10 ms, so a one-second clip becomes about a hundred columns. The result is a matrix, and almost every speech model consumes a matrix of this shape. The axes are the contract: rows are frequency bins, columns are time.

The framing trade

The window length sets the trade. A long window has narrow bins and good frequency resolution, but it blurs events in time. A short window pinpoints a click and cannot separate close tones. Speech usually settles on 20 to 25 ms, which is short enough for a consonant and long enough for a vowel’s harmonics. There is no universal answer, only an answer for the sounds you care about. A 25 ms window at 16 kHz is 400 samples, so the bin spacing is 40 Hz.

The hop sets how often a column is produced. A hop equal to the window gives no overlap and a choppy picture. Half the window is the common default, and a quarter is used when time precision matters. More overlap costs more compute and storage for a smoother image. The hop does not change the frequency resolution; only the window length does that. That mix-up is common.

An STFT and a mel spectrogram

import librosa
import numpy as np

y, sr = librosa.load("speech.wav", sr=16000)
S = np.abs(librosa.stft(y, n_fft=400, hop_length=160))
mel = librosa.feature.melspectrogram(S=S ** 2, sr=sr, n_mels=40)

print(S.shape, mel.shape)

Reading the picture

A voiced vowel (one made by vibrating the vocal folds) shows a stack of horizontal lines: the fundamental and its harmonics, spaced by the pitch. The envelope over those lines is shaped by the vocal tract, and its peaks are the formants (the bands the throat boosts). An unvoiced fricative (a hiss like s or f) looks like a band of noise with no lines. A plosive is a short vertical smudge followed by silence or a burst. Reading those patterns is most of practical audio inspection. Time runs left to right, so a syllable is a shape with a beginning and an end.

Magnitude is usually plotted on a log scale, in decibels, because the ear is logarithmic and a linear plot hides quiet detail under one loud peak. Twenty times the log base ten of the magnitude is the standard conversion. The colour map matters too: a rainbow can invent boundaries that are not there, while a single-hue ramp is easier to read. Label the axes and the units before you trust your eyes.

The mel compression

The mel scale is a perceptual remapping of frequency. Below 1000 Hz it is nearly linear. Above it, the scale compresses, because the ear resolves high frequencies less finely. A mel filterbank is a set of triangular filters spaced evenly on that scale, and applying it turns a linear spectrum into a few dozen bands. The result is smaller and more robust to small pitch shifts.

A mel spectrogram is not a spectrogram in the strict sense; it is a compressed summary of one. It is the standard input to speech recognisers and audio classifiers. It keeps the information that distinguishes sounds and discards the fine structure that varies between speakers. The number of mel bands is a hyperparameter (a setting you choose before training), usually between 40 and 80. More bands keep more detail and need more data to train. The choice is a trade between resolution and statistical efficiency.