Speech and audio from the ground up
Speaker and language identity
Who is speaking, and in which language? Both answers start with an embedding, but the threshold that turns a score into a decision belongs to the application, not the model.
Two questions, one embedding
Speaker recognition asks who is speaking. Verification compares one voice to one enrolled voice and answers same or different. Identification compares one voice to a closed set and picks the closest. Language identification asks which language is being spoken, usually to route the audio to the right recogniser. The three tasks share a shape: turn a recording of any length into a fixed-length vector, then compare vectors or train a classifier on them. The recording can be one second or one hour; the vector is the same size.
The embedding is the important object. A good speaker embedding keeps identity and discards what was said, the channel, the room and the microphone. A good language embedding keeps the sound system and rhythm that tell languages apart and discards the speaker. Neither comes for free. A model trained on one dataset learns that dataset’s channels as a shortcut, and a score that looks great in the lab can collapse on a phone. The same recording can be easy for one task and hard for the other.
Embeddings and scoring
Classical systems used i-vectors; modern ones use x-vectors from a TDNN, or ECAPA-TDNN embeddings. Those are names of network designs, and the details matter less than the output. Frames are pooled, usually with statistics pooling that keeps a mean and a standard deviation, and the result is a few hundred numbers. Verification is then a cosine similarity or a PLDA score (a way to weigh evidence across sessions). Scores are often normalised against a cohort of impostors (other voices used as a reference). The cohort is what makes a score comparable across trials. The embedding is not a transcript of the voice, and it should not be treated as one.
Enrolment matters as much as the model. One recording gives a noisy centroid (an average point); several recordings from different sessions give a better one, and a running average lets the profile improve over time. Short test clips are the hard case, because there is less data to average. Channel mismatch is the other: a voice enrolled on a laptop and tested on a conference phone is a different distribution, and the threshold has to survive it.
Verification with SpeechBrain
from speechbrain.inference.speaker import SpeakerRecognition
verification = SpeakerRecognition.from_hparams(
source="speechbrain/spkrec-ecapa-voxceleb",
savedir="pretrained_models/spkrec-ecapa-voxceleb",
)
score, same = verification.verify_files("enrol.wav", "test.wav")
print(round(score.item(), 3), bool(same))An equal-error threshold
Run the sweep and find the equal-error rate. Then move the impostor mean up by a tenth, as a new channel would, and watch both the EER and the threshold move.
Thresholds, EER and the cost of mistakes
A verification system returns a score, not a decision. The threshold turns one into the other, and it belongs to the application. A bank wants a low false-accept rate and will tolerate more false rejects; a phone unlock wants the opposite. The equal-error rate (EER) is the point where the two error rates meet, which makes it a convenient summary and a poor operating point. It says nothing about the cost of either mistake. Two systems with the same EER can behave very differently at the threshold you need.
Report the full curve, or at least the false-accept and false-reject rates at the threshold you ship. Calibrate the scores so a number means the same thing across models and settings, and re-calibrate when the channel changes. A threshold copied from a paper is a threshold measured on someone else’s microphones. The test set should include the awkward cases: short clips, noise, several languages and both sexes in fair numbers.
A voice embedding is biometric data. Collect it with consent, keep it only as long as the task needs it, and test for replay and synthetic speech. A speaker score is evidence about a recording, not proof of who was in the room.
Language identification is a different problem
Language identification uses different cues. Rhythm and phonotactics (which sound sequences are allowed) are strong for related languages; word lists and writing systems are strong for text but unavailable in audio. A letter or phoneme n-gram model (one that counts short sequences) is a solid baseline, and a neural encoder trained on many languages is the modern one. Short clips are the hard case: a few seconds may contain one borrowed word and no special sound. Code-switching breaks the idea that a clip has one label.
Related languages are where the errors cluster: Spanish and Portuguese, Hindi and Urdu, Malay and Indonesian. The output should be a ranked list with calibrated scores, not a single label, because the downstream recogniser can try the top two. Treat language identification as a router with a fallback, not the final word. A wrong route is cheap to recover from if the system notices, and expensive if it does not. Keep the second-best language and its score in the logs.
A character-trigram language identifier
Run the classifier and read the margin for each test sentence. Then add a Portuguese sentence and see which of the three languages it is confused with.
Check what you learned
One question at a time. A wrong answer explains itself — take another go.
Part 1 of 5
How does speaker verification differ from identification?