EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

baseline
The score you compare a change against, usually from the version already in use.
confidence interval
A range around a measured score that shows how much it could move if you tested a different sample.
embedding
A list of numbers that represents a piece of text, placed so texts with similar meaning sit close together.
exact match
A score of one only when an answer matches the expected text after ignoring case, punctuation and spacing, and zero otherwise.
golden set
A fixed list of questions with the answers or properties you will accept, used to compare changes.
groundedness
The share of claims made by a model that the sources you supplied actually support.
judge
A model asked to score or compare answers against a rubric, used when fixed rules cannot measure quality.
recall
The share of everything that should have been found that was actually found.
regression
A drop in quality against a result you had already accepted, caught by rerunning the same tests.
rubric
A written list of criteria with descriptions of what a good, acceptable or failing answer looks like.
semantic similarity
A score for how close two texts are in meaning, usually by comparing their embeddings.
threshold
The cut-off score at which you accept a result rather than reject it or send it for review.
token F1
A score that combines two overlap rates, one for covering the expected text and one for avoiding extra text, so both must be good.

Evaluating AI systems

Semantic similarity

Comparing embeddings catches paraphrases that exact match misses — and quietly passes contradictions. Used with a threshold and a second check, it is the cheapest useful judge.

Part 4 of 10

Intermediate14 min readBack to Courses

The idea, and its appeal

Embed both the expected answer and the reply with the same model, compute the cosine similarity, and compare it with a threshold. Above it, the answers are treated as equivalent. It is one API call, needs no reference beyond the golden answer, and tolerates any wording.

For short factual answers it works well, because a correct paraphrase lands close to the reference and an unrelated sentence does not. It is also the metric most often over-trusted.

A worked pair: “five business days” against “within a week” scores around 0.82 with a typical model, because the durations are close in meaning. The same reference against “support replies in two days” scores around 0.12, because the topic is different. A threshold near 0.8 accepts the paraphrase and rejects the unrelated answer, which is exactly the behaviour the metric promises.

The contradiction problem

“Refunds take five days” and “Refunds do not take five days” are about the same thing, use almost the same words, and score very high. Embeddings capture topic, and a contradiction is on the same topic.

That is not a flaw you can tune away with a better model. It follows from what an embedding is: a position in a space organised by meaning-in-context, not a truth value. A similarity metric can tell you the answer is about the right subject; it cannot tell you the subject was affirmed.

A model trained on topic similarity has no reason to place “not” far away. The negation is one token among many, and the sentence remains about refunds, days and policy. The vector moves slightly, not enough to cross a threshold. That is why the failure is systematic rather than occasional: every contradiction phrased in the same vocabulary will pass.

Scoring a reply

from langchain_openai import OpenAIEmbeddings
from math import dist

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
expected = embeddings.embed_query("five business days")
reply = embeddings.embed_query(answer)

similarity = sum(a * b for a, b in zip(expected, reply))
print(similarity)          # compare with a calibrated threshold

A threshold over toy vectors

Score each reply, then move the threshold to 0.95 and to 0.5. Count how many correct replies are rejected and how many wrong ones are accepted at each value.

Output
Click "Run" to execute your code

Thresholds are calibrated, not guessed

A threshold of 0.8 means nothing on its own. The same value is strict for one model and loose for another, and it depends on the length of the answers: short texts have noisier similarities than long ones.

Calibrate on labelled pairs. Take twenty replies you have judged correct and twenty you have judged wrong, plot their scores, and pick the value that separates them. If no value separates them, the metric cannot do this job and you need a different one.

Calibration is a small experiment, not a guess. If the correct replies run from 0.72 to 0.95 and the wrong ones from 0.30 to 0.88, there is no clean cut: a threshold of 0.86 accepts three wrong answers to keep most of the right ones. That overlap is the finding, and it may mean the metric is not good enough for this case.

Never compare similarities from two different embedding models. The numbers look alike and mean different things, so a threshold tuned on one silently mislabels the other.

Similarity is not entailment

Similarity asks whether two texts are about the same thing. Entailment asks whether one follows from the other. A contradiction is maximally similar and not entailed at all, which is why a second check is needed.

A small natural-language-inference model, or a judge with a yes/no rubric, answers the entailment question directly. Run it only on the pairs that pass similarity: the cheap metric filters the obvious misses, and the expensive one adjudicates the near neighbours where contradictions live.

Where it earns its place

Similarity is a good first pass over a large set: it is cheap, deterministic and catches the replies that are nowhere near the answer. The cases near the threshold are the ones worth sending to a model-based judge or a person.

Used that way it is a filter, not a verdict. The expensive judgement is reserved for the ambiguous middle, which is exactly where human and model attention are worth spending.

Report the model name and the threshold beside the score. A similarity number without them cannot be compared with last month’s, and a reader cannot tell whether a change in the score came from the system or from a new embedding model.

A contradiction that scores high

Compare the two replies with the expected answer. Notice that the negation is almost as similar as the paraphrase, then add a simple negation check that catches it.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does semantic similarity catch that exact match misses?