EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

approximate nearest neighbour (ANN)
A search that checks only part of the index, so it is fast but may miss some of the true closest matches.
collection
A named group of vectors that share one configuration and are searched together.
cosine similarity
The dot product of two unit-length arrows, from 1 when they point the same way to -1 when they oppose.
dot product
Multiply matching numbers in two lists and add the results; it grows with both length and agreement.
embedding
A list of numbers that places a piece of text in a space where similar meanings sit close together.
embedding model
The trained program that turns text into vectors, with one fixed output width.
hybrid search
Running a word-matching search and a meaning-based search together, then combining their ranked lists into one.
index
A structure built ahead of time so the database can find nearby items without comparing against every one.
magnitude
The length of a vector, found by squaring its numbers, adding them and taking the square root.
metadata filter
A rule that keeps only the items whose stored facts, such as date or owner, match before or during the search.
model card
A short document describing a trained system: its data, its scores and its known limits.
normalised vector
A vector divided by its own length, so it has length one and only direction remains.
quantisation
Storing each number with less precision, which shrinks memory and usually costs a little accuracy.
recall
The share of the truly closest items that a search actually returned.
reranking
Scoring a short list of results again with a slower, more careful method and putting the best first.
similarity function
A rule that scores two vectors and returns a number saying how alike they are.
top-k
The shortlist a search returns: its best few results, cut off at a number chosen in advance.
training loss
The average error an embedding model makes on the text pairs it is trained to match.
unit length
A vector whose magnitude is exactly 1, which makes a dot product read as pure agreement.
vector
An ordered list of numbers, where each position is one measured property.

Vector databases from the ground up

Capstone: a small index with recall measurement

Build a flat index and a partitioned index over the same vectors, then measure recall@k and the work each one does. The number that matters is the one you measured on your own queries.

Part 10 of 10

Advanced18 min readBack to Courses

The harness comes first

Before tuning anything, build the answer key. Flat search over the same corpus gives the true top k for every query. Recall@k is the fraction of those neighbours an approximate index returned. Without that number, “the index feels good” is the whole evaluation. The harness is not a one-off script. It is the thing that tells you a setting change helped rather than moved the problem.

Count work as well as time. Distance computations are the unit that survives a faster machine, and a candidate count makes the recall curve readable: this much recall for this much scanning. Wall-clock latency still matters for the product, but it is a noisy measurement and a poor thing to tune against on a laptop.

The harness also fixes the vocabulary for the rest of the chapter. Recall is about the answer set; latency is about the clock; work is about the number of distances computed. Keeping them separate stops a fast index with terrible recall from looking like an improvement. It also stops a slow index with perfect recall from looking like a failure. Three numbers, three questions, and a report that can be read without a glossary.

Read the curve, not the point

Sweep the knob and plot recall against candidates scanned. The useful shape is a steep rise and then a plateau. The knee is where the next doubling of work buys almost nothing. Pick a point a little past the knee, not at the maximum. Maximum recall is always available and always costs the whole corpus.

A single query proves nothing. Forty or eighty queries drawn from real traffic will show the tail: the query whose answer lives in the partition you did not probe. The tail is what users complain about. It is the part a synthetic benchmark with uniform random queries is least likely to reproduce.

Ground truth from flat, recall from ANN

import faiss

flat = faiss.IndexFlatIP(dim)
flat.add(vectors)
truth = flat.search(queries, k=5)[1]

ann = faiss.index_factory(dim, "IVF256,Flat")
ann.train(vectors)
ann.add(vectors)
ann.nprobe = 8
found = ann.search(queries, k=5)[1]

recall = (found == truth).any(axis=1).mean()

Measure recall against the exact answer

Run the harness, then add a graph index to the comparison. Keep the same queries so the three numbers are measured on the same ground truth.

Output
Click "Run" to execute your code

What to ship

Start flat if the corpus fits and the latency budget allows it. When it does not, ship the approximate index with the settings you measured. Keep an exact path for a small sample of traffic so drift is visible before a customer reports it. The exact path can be one percent of traffic and still catch a bad deploy the same day.

Record the index type, the metric, the embedding model and the settings next to the collection. Six months from now the only person who can explain a recall drop is the one who can see what changed. That record belongs in the migration history, not in a wiki page that was last edited before the current team joined.

Keep one percent of traffic on exact search and compare the two result sets. It is the cheapest recall monitor you will ever build, and it catches a bad parameter change the same day.

The questions that remain

A production index also needs a rebuild path, a way to backfill vectors (fill in old ones) when the model changes, and a delete that is visible everywhere. Those are the chapters before this one, applied to a real corpus rather than a synthetic one. None of them is hard on its own. The difficulty is that they share the same collection and the same maintenance window.

The capstone is small on purpose so that every number can be inspected. Scale changes the constants, not the method: ground truth, a sweep, a knee, and a monitor that keeps the promise honest. If you can explain this harness to a colleague, you can explain why the production index is set up the way it is.

If you take one habit from this course, take the sample. Every approximate system should be able to answer “compared to what?” without a rebuild, a benchmark harness or a vendor. A sampled exact query is that answer, and it costs almost nothing to keep. The day the index changes behaviour, the sample is the difference between a rumour and a measurement.

Sweep the knob and pick the knee

Run the table, then choose the smallest nprobe whose recall is at least 0.95. Change the number of lists to 8 and 32 and see how the knee moves.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Where does the ground truth for recall come from?