Speech and audio from the ground up
Text to speech
A synthesizer has to decide what the words say, how they are pronounced, how the sentence moves, and only then how the samples sound.
Text is not sound
Text to speech starts before any audio exists. Numbers, dates, currency, abbreviations and URLs have to be expanded into words, and the expansion depends on context: “2026” is a year, “2/6” is a date or a fraction. Homographs (words spelled the same but said differently) need a decision too, since “read” and “lead” change sound with tense or meaning. A front-end that gets this wrong produces a confident, fluent, wrong sentence. No vocoder (the part that makes the waveform) can repair the wrong word. The front-end is where most awkward failures begin.
Normalisation is language-specific and full of exceptions. A phone number is read digit by digit, a year in pairs, a price with its currency. Punctuation becomes pauses, and capitalisation (upper-case letters) sometimes becomes emphasis. The rules are unglamorous and they dominate perceived quality. Listeners notice a misread date at once and forgive a slightly metallic vowel. Build the normaliser from real text, not from invented examples. Log the sentences it changes so a reviewer can audit them.
From spelling to sound
Grapheme-to-phoneme conversion maps spelling to sounds. English is the hard case: “though”, “through”, “tough” and “bough” share the letters ough and pronounce them four different ways. A pronunciation dictionary covers common words exactly, and a learned or rule-based fallback handles the rest. The fallback matters most for names, which are exactly the words a dictionary is least likely to contain. A system that reads every common word and stumbles on the customer’s name feels broken. A dictionary is a product decision as much as a linguistic one.
Sounds (phonemes) are not the only unit. Some systems use graphemes directly, some use syllables, and some use a learned subword inventory (a set of reusable pieces). The choice affects how much pronunciation knowledge is explicit and how easy the system is to correct. A phoneme inventory is compact and portable; a grapheme system needs less linguistic work but more data. For languages with shallow orthography (spelling that closely matches sound) the difference is small. For English it is the difference between a dictionary and a guess.
Synthesis with a library
from TTS.api import TTS
tts = TTS(model_name="tts_models/en/ljspeech/tacotron2-DDC")
tts.tts_to_file(text="Hello from the compiler.", file_path="out.wav")A pronunciation lookup with a fallback
Run the converter and find the words it had to spell out. Then add one of them to the lexicon and confirm the output changes.
Prosody carries the structure
Prosody is the part that makes speech sound like speech. Duration decides which syllable is long, pitch decides the melody, and pauses decide the phrasing. A question rises at the end. A list has a small rise on each item and a fall on the last. Emphasis stretches and raises the stressed word. The same phonemes with flat prosody are intelligible and exhausting, because the listener has to rebuild the structure the voice refused to give. Read the sentence aloud and mark where you breathe; that is the target.
Prosody is also where meaning lives. “I did not say that” has five readings depending on stress, and a synthesizer that picks one at random can be correct and misleading. Predicting it needs syntax, information structure and sometimes the wider context. Most systems get the defaults right and the exceptions wrong, so a good TTS demo uses neutral sentences. Test it on a correction, a question and a list before believing the demo.
Listen to the same sentence three times before judging a voice. The first pass hears the content, the second hears the prosody, and only the third hears the artefacts. Most demo evaluations stop at the first.
The acoustic model and the vocoder
The acoustic model turns phonemes and prosody into an acoustic representation, and the vocoder turns that into a waveform. Concatenative systems stitch recorded units together; parametric systems generate from a model; neural systems predict a spectrogram and hand it to a vocoder. The vocoder is the part that turns a spectrogram back into samples, and it is where the metallic artefacts usually live. A good vocoder is not optional: the cleanest spectrogram sounds bad through a poor one. The split lets each part be improved and tested on its own.
Testing is the weak spot. Mean opinion score asks listeners to rate naturalness on a five-point scale, and it is expensive, noisy and hard to reproduce. Word error rate from a recogniser is cheaper but measures being understood, not quality. AB tests answer a product question rather than a scientific one. Report the listeners, the sentences and the scale, and treat a difference smaller than the noise as no difference. A demo is not an evaluation.
A prosody schedule
Run the schedule and read the pitch contour. Then put the stress on the first word and see how the melody changes without any new audio.
Check what you learned
One question at a time. A wrong answer explains itself — take another go.
Part 1 of 5
Why is text normalisation needed before synthesis?