Speech and audio from the ground up
Sound as samples
A recording is a list of numbers taken at a fixed rate. The rate and the bit depth decide what the list can represent, and what it silently invents.
Pressure over time
A sound is a pressure wave. A microphone turns the changing pressure into a changing voltage. An analogue-to-digital converter (the chip that turns a voltage into numbers) then measures that voltage at regular intervals. Each number is one sample: the amplitude (how far the pressure has moved) at one instant. Play the numbers back through a speaker at the same rate and the pressure wave returns, close enough that the ear cannot tell the difference. The whole of digital audio is making that sentence true.
The sample rate is how many numbers are taken each second. A telephone line uses 8000, a compact disc 44100, and speech models usually 16000. The rate is not a quality setting in the abstract. It is a limit on the highest frequency the recording can represent. Everything above that limit is not merely lost; it is folded back into the signal as something else, a false lower tone. That folding is the first fact every audio engineer learns.
Sampling and the Nyquist limit
The Nyquist theorem says a sample rate must be greater than twice the highest frequency you want to keep. At 16000 samples per second, the ceiling is 8000 Hz. Human speech carries most of the energy that makes it understandable below 4000 Hz. So 8000 is enough for a telephone, and 16000 is comfortable for a recogniser (the system that turns audio into words). Music needs more because cymbals and consonants live higher. The theorem is a statement about the signal, not the equipment: a perfect sampler still aliases.
Sample too slowly and a high tone appears as a lower one. A 5000 Hz tone sampled at 8000 Hz comes back at 3000 Hz, because the samples cannot tell the two apart. The false frequency is called an alias, and no later processing can tell it from a real 3000 Hz tone. The fix is a low-pass filter (one that keeps low tones and removes high ones) before sampling, not a smarter algorithm after. Anti-aliasing is a hardware duty.
Reading a file into samples
import librosa
y, sr = librosa.load("speech.wav", sr=16000, mono=True)
print(y.shape, sr, round(librosa.get_duration(y=y, sr=sr), 2))A sine wave in a list
Run the statistics, then double the frequency and confirm the zero crossings double while the RMS stays the same.
Quantisation and its hiss
Quantisation is the other axis. Each sample is stored with a fixed number of bits, so the amplitude is rounded to the nearest level those bits can store. Sixteen bits give 65536 levels and a signal-to-noise ratio (how loud the signal is compared with the noise) of about 96 decibels. Eight bits give 256 levels and about 48. The error is heard as a quiet hiss, and it is worst when the signal is quiet. The rule of thumb is six decibels per bit. The quiet parts of speech are exactly where that hiss sits.
Dither is the counter-intuitive fix: add a tiny amount of noise before rounding. The added noise breaks the link between the error and the signal. That link is what makes quantisation sound like distortion rather than hiss. The total noise rises slightly and the unpleasant artefact disappears. For speech recognition, the difference rarely matters; for music, it separates acceptable from not. The ear is more sensitive to pattern than to level.
Check the sample rate before anything else. A 44.1 kHz file treated as 16 kHz plays fast and shifts every formant; the model will look broken when the metadata is what is wrong.
Frames are the unit of analysis
Speech is not stationary (its statistics keep changing). A vowel holds a steady shape for tens of milliseconds, and a plosive (a burst like p or t) changes in five. Analysing a whole sentence with one Fourier transform (a breakdown into sine waves) smears all of that together. The standard compromise is to cut the signal into frames of 20 to 25 milliseconds, with a hop of 10 milliseconds. Each frame is then analysed as if it were steady. At 16000 Hz a 25 ms frame is 400 samples and a 10 ms hop is 160.
Frames overlap because a feature that lands on a boundary should not vanish. A window function tapers each frame to zero at its edges. Without that taper, the cut itself creates a sudden jump that looks like high-frequency energy. The Hann window is the default. The result is a sequence of short spectra, which is the input to almost every speech model in this course. Overlap and windowing are not cosmetic; they decide what the spectrum says.
The tone that comes back wrong
Run both recordings and compare the estimated frequency. Then try 3500 Hz at 8000 and explain why that one survives.
Check what you learned
One question at a time. A wrong answer explains itself — take another go.
Part 1 of 5
What is one sample?