Capstone: features and word error rate

Two halves that meet at a number

The capstone has two halves that meet at a number. The first is a feature extractor: audio in, a matrix of coefficients out. The second is an error report: references and hypotheses (the correct and the produced transcripts) in, a table of mistakes and a corpus rate out. Neither is impressive alone. Together they are the smallest complete speech pipeline, and every production system is a larger version of the same shape. The interfaces between the halves are where most integration bugs live. The shape is the lesson; the size is a small matter.

The extractor is tiny on purpose: one channel, 8 kHz, 256-sample frames, 20 mel bands and 13 cepstral coefficients (a compact summary of the spectrum). The numbers are small enough to print and large enough to be real. The report is honest on purpose: it shows the corpus rate, each utterance and the worst case. A capstone is not a demo; it is a claim you can check. Keep the setup next to the output so the claim can be reproduced.

The extractor, step by step

The pipeline is the one from the course, in order. Frame the signal with overlap, apply a Hann window, take the magnitude spectrum, apply a mel filterbank, take the log, and apply the DCT. The DCT (a cosine transform) packs the shape into few numbers. Each step has a setting, and every setting belongs in the setup. The output is a matrix with one row per frame and one column per coefficient. Print the shape before the values; a transposed matrix is the most common bug in audio code. A shape of (frames, coefficients) is the convention most libraries follow.

Two details decide whether the numbers mean anything. The first is consistency: the same sample rate, window and mel count must be used wherever the features are compared. The second is scale: log energies can be negative, and a model may expect a particular normalisation. Standardise per utterance or per dataset, but write down which. A feature matrix is not self-describing, and a future reader will not guess the settings from the shape.

The same chain in two libraries

import torchaudio
from jiwer import wer

waveform, sr = torchaudio.load("speech.wav")
mel = torchaudio.transforms.MelSpectrogram(sample_rate=sr, n_mels=40)(waveform)
print(mel.shape)
print(round(wer("the quick brown fox", "the quik brown box"), 3))

The report refuses a single number

The report starts with the corpus rate and then refuses to stop there. For each utterance it prints the reference length, the edits and the rate. It totals substitutions, deletions and insertions separately, because a system that deletes endings and one that inserts filler need different fixes. It names the worst utterance, since that is where the next hour of work should go. A table of numbers is more useful than a single score. The worst case is usually more informative than the mean.

Then slice. Group the utterances by speaker, channel or noise level, and report the worst group with its size. A group of three is a hint; a group of three hundred is a finding. Keep the references and hypotheses as files so the report can be rerun after a change. The point of the report is not to certify the system; it is to make the next change measurable. A baseline you cannot rerun is a memory, not a baseline.

Reproducibility and limits

Reproducibility is a feature of the pipeline, not an afterthought. Pin the sample rate, the frame length, the hop, the mel count and the normalisation in one place, and print them with the result. A different NumPy version should not change a number that matters. Keep a tiny fixed example with known output so a refactor that silently changes the features fails a test. The fixture is worth more than a long comment.

The limits are part of the result. This extractor has no delta features (the change from frame to frame), no voice activity detection and no noise robustness. The report has a handful of utterances and no confidence intervals (a range around each estimate). Say so. A small system that states its scope is more trustworthy than a large one that hides it, and the next engineer can see exactly which assumption to revisit. The capstone is finished when the limits are as clear as the numbers. Write the limits before the summary, so they cannot be forgotten.