Features for speech

Why raw samples are a poor feature

A second of 16 kHz audio is 16000 numbers. Most of them are redundant: neighbouring samples are almost equal, and the phase of a waveform matters little to the ear. A classifier trained on raw samples has to learn the Fourier transform, the mel scale and the statistics of speakers from data. Features do that work once, in code, and shrink the input by orders of magnitude. The cost is that a bad feature throws detail away for good.

The goal is not to lose information for its own sake. It is to keep what tells one sound from another and discard what changes between two recordings of the same sound. Loudness, channel gain, speaking rate and pitch vary; the shape of the spectrum carries the message. A good feature is a bet about which variation is signal and which is nuisance. That bet is testable, and it is usually tested by training two models.

The mel filterbank

The mel filterbank is the first compression. Frequencies are mapped onto a perceptual scale, and a set of triangular filters overlaps across it. Each filter sums the energy in its band, so a 129-bin spectrum becomes 40 numbers. The triangles are wider at high frequencies, matching the ear’s coarser resolution there. The filter edges are usually chosen to cover 0 to 8000 Hz at 16 kHz. The sum is a weighted average, not a pick of one bin.

The log comes next, and it is not cosmetic. Loudness is perceived roughly logarithmically, so a quiet consonant and a loud vowel differ by a large linear factor and a small log difference. Taking the log compresses that range. It also turns multiplication by channel gain into addition, which a normalisation step can remove. Log-mel energies are the workhorse feature of modern speech systems. They are also close to what a cochlea reports.

MFCCs from a library

import librosa

y, sr = librosa.load("speech.wav", sr=16000)
mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13, n_fft=400, hop_length=160)
delta = librosa.feature.delta(mfcc)

print(mfcc.shape, delta.shape)

The cepstrum and MFCCs

Mel bands overlap, so neighbouring energies are correlated. The discrete cosine transform (DCT) decorrelates them and concentrates the shape into a few coefficients. Keep the first 13 and you have MFCCs: mel-frequency cepstral coefficients. The first coefficient is overall level, and the rest describe progressively finer spectral shape. The name is scary; the operation is a rotation followed by a truncation. It is a compression, and like all compression it assumes the discarded part is noise.

MFCCs were the standard for decades and are still a strong baseline. Their weakness is that the truncation and the decorrelation discard detail that a large model could use. So mel spectrograms became the default input for neural recognisers. The choice is not ideological: MFCCs are compact and robust on small data, log-mel keeps more detail and rewards more data. Measure both on your own test set. A feature is only better relative to a model and a dataset.

Normalisation and context

Cepstral mean and variance normalisation removes the average channel. For each coefficient, subtract the utterance mean and divide by the standard deviation. A different microphone adds a constant in the log domain, and the subtraction deletes it. The same operation makes features comparable (similar enough to compare) across speakers and recordings, as long as it is applied consistently at training and inference. Applying it per utterance is a choice; per dataset is another.

Static features say what the spectrum is at one moment. Deltas say how it is changing, and delta-deltas say how the change is changing. Stacking a few frames of context gives the model a short window without a recurrent network (one that carries state from step to step). None of these adds information that was not in the audio; they make the information easier to extract. The pipeline is a set of bets, and every bet should be checked against a baseline.