The ASR pipeline

Four stages and a search

A recogniser is a chain of four parts. Audio becomes features, the features become a sequence of acoustic scores, a decoder searches for the most likely text, and the text is formatted. Each part has its own failure modes, and the word error rate (the share of words that differ from the reference) at the end hides which one failed. When accuracy drops, the first job is to find the stage where the right answer was still present. An error report should therefore include examples, not only a number.

The acoustic model does not output words. It outputs a distribution (a spread of scores) over subword units, usually sounds (phonemes), characters or sentence pieces, for every frame. Those units are then assembled by the decoder using a pronunciation lexicon (a list of how words sound) and a language model. End-to-end systems collapse the stages into one network. The same three questions remain: what sounds were said, what words could they be, and what is likely in this language. The answers can be learned together or supplied separately.

The acoustic model

The model sees a stack of frames and must decide what is being said now. Recurrent networks and transformers add context, because a single 25 ms frame is ambiguous: many sounds share the same short spectrum. The output is usually a probability per unit per frame, and the alignment between frames and units is unknown. That unknown alignment is the central technical problem, and CTC and attention solve it in different ways. CTC is trained with a loss that sums over alignments; an attention model learns its alignment as it generates.

CTC adds a blank symbol and lets the model emit nothing while it waits. Repeated units collapse, so a long vowel does not become a long string of the same letter. The loss sums over every alignment that collapses to the target, which avoids needing frame-level labels. Attention models instead learn to look at the relevant frames while generating tokens one at a time. CTC is simpler and streams naturally; attention is more flexible and often more accurate.

A recogniser in three lines

import whisper

model = whisper.load_model("base")
result = model.transcribe("speech.wav", language="en")
print(result["text"])

The lexicon and the language model

The lexicon lists how each word is pronounced, usually as a sequence of phonemes. The language model says which word sequences are plausible, and the decoder combines the two scores with the acoustic score. The weighting between them is a real hyperparameter (a setting you choose). Too much language model and the system hears what it expects. Too little and it produces phonetically valid nonsense. Beam search keeps the best few guesses alive instead of committing early. A wider beam costs time and buys accuracy, with diminishing returns.

End-to-end models fold the lexicon and language model into the network, which removes a lot of work and makes low-resource languages easier. The trade is control. A separate lexicon can be corrected for a name without retraining, and a language model can be swapped for a domain. A single network is harder to inspect and harder to fix in one place. Most production systems choose based on which kind of correction they expect to need.

Streaming and latency

Streaming recognition produces words while the speaker is still talking. The model can only use past audio, so it needs a causal design (one that never looks ahead) or a bounded lookahead, and the decoder must revise partial guesses. A larger lookahead usually improves accuracy and always adds latency. The product decision is how much delay a user will tolerate, and the engineering decision is how to spend that budget on context. A dictation tool and a meeting transcriber can use the same model with different budgets.

Latency is not only the model. Feature extraction buffers a frame, the network runs a forward pass, the decoder waits for enough tokens, and the display updates. Each stage adds a few milliseconds that are invisible in a benchmark and audible in a conversation. Measure end-to-end time on the target device, not on a workstation. The slowest part is often the one nobody measured. Profile before tuning, and profile on the device that will run it.