Speech and audio from the ground up
Samples, spectra and the words they carry
- Intermediate
- 10 chapters
- 157 min read
How a recording becomes a transcript and a transcript becomes a voice: sampling and the Nyquist limit, the Fourier view, spectrograms and their trade-offs, mel and cepstral features, the ASR pipeline, word error rate, and text to speech. Every transform is written out in plain Python, and the capstone builds a tiny feature extractor and an honest error report.
What You'll Learn
- Explain sampling, aliasing and quantisation
- Read a spectrum and choose a window for the question
- Interpret a spectrogram and its time-frequency trade-off
- Compute mel filterbank energies and MFCCs
- Describe the stages of an ASR system and what each one decides
- Compute word error rate and explain what it hides
- Design a TTS front-end and a simple prosody schedule
- Build a feature extractor and a per-utterance error report
Prerequisites
- Comfortable with Python functions, lists and loops
- Basic trigonometry and logarithms
- No audio library is needed — every example is plain Python
Vocabulary
The terms the chapters assume, in plain words.
- aliasing
- The false low tones that appear when a signal is sampled too slowly for the frequencies it contains.
- amplitude
- How far a wave swings from its resting point; the loudness of a sound.
- ASR (automatic speech recognition)
- The task of turning recorded speech into written words by machine.
- Fourier transform
- A calculation that splits a signal into the pure frequencies that add up to it.
- frequency
- How many times a wave repeats each second; one hertz is one repeat per second.
- mel scale
- A pitch mapping built to match how human hearing spaces low sounds further apart than high ones.
- MFCC
- A compact set of numbers that describes the shape of a short sound's energy across the mel scale.
- phoneme
- The smallest unit of sound that can change a word's meaning in a language.
- sample rate
- How many measurements of a continuous signal are taken each second.
- spectrogram
- A picture of how a signal's energy at each frequency changes over time.
- streaming
- Processing audio as it arrives, in small pieces, instead of waiting for the whole recording.
- waveform
- A plot of a signal's amplitude over time.
- window
- A short slice of a signal, faded at the edges, that one analysis step examines.
- word error rate
- The number of word insertions, deletions and swaps needed to turn a transcript into the correct text, divided by the number of correct words.
Course Content
- 01Sound as samplesNot started
A recording is a list of numbers taken at a fixed rate. The rate and the bit depth decide what the list can represent, and what it silently invents.
- 02The Fourier viewNot started
Every signal is a sum of sinusoids, and the Fourier transform says how much of each. The catch is that a finite recording can only answer the question approximately.
- 03SpectrogramsNot started
Stack a spectrum for every frame and time reappears. The spectrogram is the picture that makes speech readable, once you know what the trade-offs did to it.
- 04Features for speechNot started
Raw samples are too many and too literal. A feature pipeline decides what to keep: mel bands, log energies, cepstral coefficients and the deltas between them.
- 05The ASR pipelineNot started
Audio to features, features to acoustic scores, scores to words. Each stage has a different job, and the word error rate at the end hides which one went wrong.
- 06Word error rateNot started
One number for a transcript: the edits needed to turn the hypothesis into the reference, divided by the reference length. Useful, comparable, and easy to over-read.
- 07Text to speechNot started
A synthesizer has to decide what the words say, how they are pronounced, how the sentence moves, and only then how the samples sound.
- 08Streaming speech recognitionNot started
A live recogniser cannot wait for the sentence to end. Chunk size, lookahead and endpointing decide how quickly it commits, and every one of them is a trade against accuracy.
- 09Speaker and language identityNot started
Who is speaking, and in which language? Both answers start with an embedding, but the threshold that turns a score into a decision belongs to the application, not the model.
- 10Capstone: features and word error rateNot started
Build the whole chain once: samples to frames to mel to cepstra, then references and hypotheses to an honest table of mistakes.