EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

BM25
A word-matching score that ranks a document higher when it contains rare query words, with a cap on repetition.
chunk
A short passage cut from a longer document, small enough to retrieve and cite on its own.
citation
A pointer from an answer back to the exact passage it came from, so a reader can check it.
context window
The fixed amount of text a model can read in one request, counting the question and everything retrieved.
corpus
The whole collection of documents a system searches.
cosine similarity
A score from -1 to 1 for how closely two arrows point in the same direction, ignoring their lengths.
dot product
Multiply matching numbers in two lists and add the results; a larger total means closer agreement.
embedding
A list of numbers that places a piece of text in a space where similar meanings sit close together.
grounded answer
A reply built only from the passages that were retrieved, so its claims can be traced to them.
index
A structure built ahead of time so the system can find relevant passages without reading everything.
lexical search
Finding passages by the words they contain, matching the query's terms rather than its meaning.
magnitude
The length of a vector, found by squaring its numbers, adding them and taking the square root.
reranking
Scoring a short list of results again with a slower, more careful method and putting the best first.
retrieval
The step that selects passages from the corpus to put in front of the model.
top-k
The shortlist a search returns: its best few results, cut off at a number chosen in advance.
vector
An ordered list of numbers, where each position is one measured property.

RAG from the ground up

Evaluating retrieval

You cannot improve what you do not measure. Recall finds out whether the right passage was even fetched; MRR tells you how high it ranked. Together they explain most bad answers.

Part 7 of 10

Advanced16 min readBack to Courses

A labelled set of questions

Evaluation starts with questions you know the answers to. Twenty is enough to begin: take real questions, find the passage that answers each one, and record its id. That list is the ground truth everything else is measured against.

It does not need to be large to be useful. It needs to be representative — the questions people actually ask, including the awkward ones — and it needs to be reviewed when the corpus changes.

The labelling is the work. For each question, search the corpus by hand, read the candidates, and write down every passage that would support a correct answer. One question can have three valid passages; the metric should accept any of them. Include a few questions the corpus cannot answer, because refusal is part of retrieval quality and an empty expected list is a legitimate label.

Recall first, then rank

Recall at k asks a yes-or-no question: was the correct passage among the k retrieved? If it was not, no prompt and no model can recover — the answer was never in the context. That is why recall is the first metric to look at, and the one that explains the most failures.

Mean reciprocal rank answers a different question: when the correct passage was retrieved, how high did it rank? A reciprocal rank of one means it was first, one half means second, one fifth means fifth. Averaging over the set gives a single number for rank quality.

The numbers read better with an example. Twenty questions, the right passage in the top three for eighteen: recall at three is 0.9. If ten were first, four second and four third, the MRR is (10 + 2 + 1.33) divided by 20, about 0.67. Recall tells you the passage exists in the list; MRR tells you whether the prompt will actually see it near the top.

The same idea in a framework

from ragas import evaluate
from ragas.metrics import context_precision, context_recall, faithfulness

result = evaluate(
    dataset=questions_with_ground_truth,
    metrics=[context_precision, context_recall, faithfulness],
)
print(result)

Recall and MRR over three questions

Compute both metrics, then change the retrieval order so the correct passage drops to third and watch the MRR fall while recall stays the same.

Output
Click "Run" to execute your code

Not every relevant passage is equal

Sometimes two passages are relevant and one is better: the exact answer versus a paragraph that mentions the topic. Graded relevance captures that — a score of two for the passage that answers the question, one for a related one, zero for irrelevant.

Discounted cumulative gain sums those grades while penalising lower positions, and normalising by the best possible ordering gives nDCG between zero and one. It is the metric that notices a system which returns the right passages in the wrong order.

The arithmetic is small enough to do by hand. Grades 2, 1 and 0 in that order give a DCG of 2 + 0.63 + 0, about 2.63; the ideal order is the same, so nDCG is 1. Swap the first two and the DCG becomes 1 + 1.26, about 2.26, and nDCG falls to 0.86 even though the same three documents were returned. That is the ordering penalty, made visible.

Read the failures, not only the score. A recall of 0.8 tells you something is wrong; the four misses tell you whether it is chunking, the embedding model or a filter.

Evaluate what the prompt receives

It is tempting to evaluate the vector retriever in isolation because that is the easiest list to capture. But the prompt receives the fused, reranked, filtered list, and that is the list the answer depends on. Measure at the same cutoff and through the same filters, or the score describes a pipeline you do not ship.

The same rule applies to k: if the prompt takes four passages, recall at four is the number that matters; recall at twenty describes the retriever, not the answer.

Where the answer actually breaks

Three failures look identical in the output. The passage was never retrieved. The passage was retrieved but ranked below the cutoff. The passage was retrieved and ranked well, but the answer spanned two chunks and neither contains the whole thing.

Each has a different fix: better recall, better ranking, or different chunking. Measuring recall and MRR separately is what tells them apart before you start changing things.

The two-chunk failure has a signature of its own: the answer is half right, and the missing half is in the next chunk id. Retrieving neighbours, or adding a sentence window around each hit, is the fix.

Ranking with grades

Compute nDCG for the ranking, then swap the first two results and watch the score drop even though the same documents are returned.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does recall at k measure?