Streaming speech recognition

Recognition before the sentence ends

Batch speech recognition waits for the whole utterance, then decodes it. Streaming recognition emits words while the audio is still arriving, which is what a caption, an assistant or a dictation tool needs. The model sees a prefix (the start of the sentence), not the whole sentence, and it has to commit before it knows how the sentence ends. That single constraint explains most of the design. A batch transcript can be revised until it is right; a live one cannot.

There are two clocks. The audio clock is how much speech has arrived; the compute clock is how long the model takes to process it. A system is real-time if the second stays behind the first, with enough margin for a burst. Latency is the gap between a word being spoken and appearing on screen, and it is the number users feel. Accuracy on the full utterance is a different number, and it can look fine while the live experience is unusable. A system that is accurate but three seconds behind is not a captioning system.

Chunks, context and carried state

A streaming model processes chunks of frames, typically 20 to 100 milliseconds of audio, and carries state from one chunk to the next. That state is the encoder’s memory of the past: convolution caches, attention keys and values, a decoder’s previous tokens. A chunked attention model also needs a little right context, a lookahead (a peek at future frames), to resolve sounds that only make sense with what follows. The chunk size and the lookahead are the two knobs that decide latency. Smaller chunks lower latency and give the model less to work with.

A naive decoder that treats every window as a fresh utterance loses words at the boundaries, because a word can straddle two windows. Overlap is the cheap fix: process a few frames twice and merge the result. A stateful encoder is the better fix, because it never forgets the left edge. Either way, the boundary is where streaming bugs live, and a test that splices a sentence across chunks will find them.

A streaming recogniser in Vosk

import json
from vosk import KaldiRecognizer, Model

model = Model("model-en")
recognizer = KaldiRecognizer(model, 16000)
recognizer.SetWords(True)

for chunk in microphone_stream():
    if recognizer.AcceptWaveform(chunk):
        result = json.loads(recognizer.Result())
        print("final:", result["text"])
    else:
        partial = json.loads(recognizer.PartialResult())
        print("partial:", partial["partial"])

Partials, endpointing and the interface

A partial hypothesis is a guess that can still change. The word shown in grey may become a different word when the next chunk arrives, and a caption that rewrites itself is harder to read than one that waits. Good interfaces mark unstable text, commit whole phrases, and avoid moving the line under the reader’s eye. The revision rate is a quality metric, not only a cosmetic detail. Users forgive a delay more easily than a flicker.

Endpointing decides when an utterance is over. An energy or voice-activity detector marks speech, a silence timeout closes the segment, and a maximum length forces a cut before the buffer grows without bound. A short timeout cuts a speaker who pauses to think; a long one adds dead air to every reply. The hangover (the number of quiet frames kept before closing) is the setting that trades those two failures.

What streaming costs, and how to measure it

Streaming usually costs accuracy. A model that cannot see the end of the sentence has less context, and a chunked encoder is a compromise against a full-attention one. The loss is small on read speech and larger on names, numbers and code-switching (mixing two languages), which is exactly where the full model was helping. Report word error rate on the streamed output, not only on a batch transcript of the same audio. The same audio can score well in batch and badly in the stream, and the batch number is the one that gets published.

Measure four numbers: first-partial latency, finalization delay, real-time factor (processing time divided by audio time) and revision rate. Re-score the final transcript with a larger model when the utterance closes, if the product can wait. Add contextual biasing for the names the product knows, and keep the bias list outside the audio path. A streaming system is a pipeline of compromises, and each one should be visible in a dashboard rather than discovered by a user. Keep the audio, the partials and the final text together for a sample of requests, so a complaint can be reproduced.