EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

approximate nearest neighbour (ANN)
A search that checks only part of the index, so it is fast but may miss some of the true closest matches.
collection
A named group of vectors that share one configuration and are searched together.
cosine similarity
The dot product of two unit-length arrows, from 1 when they point the same way to -1 when they oppose.
dot product
Multiply matching numbers in two lists and add the results; it grows with both length and agreement.
embedding
A list of numbers that places a piece of text in a space where similar meanings sit close together.
embedding model
The trained program that turns text into vectors, with one fixed output width.
hybrid search
Running a word-matching search and a meaning-based search together, then combining their ranked lists into one.
index
A structure built ahead of time so the database can find nearby items without comparing against every one.
magnitude
The length of a vector, found by squaring its numbers, adding them and taking the square root.
metadata filter
A rule that keeps only the items whose stored facts, such as date or owner, match before or during the search.
model card
A short document describing a trained system: its data, its scores and its known limits.
normalised vector
A vector divided by its own length, so it has length one and only direction remains.
quantisation
Storing each number with less precision, which shrinks memory and usually costs a little accuracy.
recall
The share of the truly closest items that a search actually returned.
reranking
Scoring a short list of results again with a slower, more careful method and putting the best first.
similarity function
A rule that scores two vectors and returns a number saying how alike they are.
top-k
The shortlist a search returns: its best few results, cut off at a number chosen in advance.
training loss
The average error an embedding model makes on the text pairs it is trained to match.
unit length
A vector whose magnitude is exactly 1, which makes a dot product read as pure agreement.
vector
An ordered list of numbers, where each position is one measured property.

Vector databases from the ground up

Hybrid search and reranking

Dense vectors are good at meaning and bad at exact strings. Lexical search is the reverse. Run both, fuse the two lists, and rerank a shortlist with a model that reads the query and the document together.

Part 6 of 10

Intermediate16 min readBack to Courses

Where dense retrieval fails

An embedding compresses a passage into a point, and exact tokens are the first thing to go. A SKU (a product code), an error code, a surname or a version number has no useful neighbourhood. The model has never seen that string, and the nearest vector is whatever looks similar in meaning. The failure is quiet because the result still looks like a plausible document, only not the one that contains the identifier.

Lexical search, BM25 or a keyword index, has the opposite profile. It nails the rare token and misses the paraphrase. Neither is a better retriever; they fail on different queries, which is the argument for running both. The two lists are also cheap to produce, so the cost of the combination is mostly the fusion and the rerank, not the retrieval.

The same weakness shows up in identifiers that look like words. “Berlin” is a city and also a surname. A version string such as “2.4.1” splits into pieces that mean nothing to the model. Dense retrieval will return something about Germany when the user wanted the release notes, and the result will look reasonable enough to ship.

Fusion is a ranking problem

You have two ranked lists and need one. Score fusion adds normalised scores, which requires the scores to be similar in scale, and they usually are not. Rank fusion ignores the scores and uses positions. Reciprocal rank fusion (RRF) gives each document the sum of 1 / (k + rank) across lists. It is crude, and it works far more often than a carefully tuned weighted sum whose weights were fitted on a demo.

RRF is popular because it needs no calibration and survives a retriever whose scores are on a different scale. Its weakness is that it throws away confidence: a document ranked first by a wide margin counts the same as one that barely edged out second. The constant in the denominator softens the advantage of the top slot. Changing it therefore changes the winner more often than people expect.

Two retrievers and a cross-encoder

from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder

bm25 = BM25Okapi([doc.split() for doc in documents])
lexical = bm25.get_top_n(query.split(), documents, n=50)

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
scores = reranker.predict([(query, doc) for doc in shortlist])

Fuse two lists by rank

Run the fusion, then set k=0 and compare the scores. The constant softens the top slot; without it, a first place in either list dominates the sum.

Output
Click "Run" to execute your code

Reranking buys precision

A bi-encoder embeds the query and the document separately, so it can precompute every document vector and compare in milliseconds. A cross-encoder reads the pair together, which is far more accurate and far too slow to run over the corpus. The difference is information: the bi-encoder never sees the query and the document at the same time.

Retrieval is two stages: a cheap retriever produces fifty candidates, an expensive reranker orders them, and the top five reach the prompt. The reranker is where a marginal retriever can be rescued, and where the latency budget is usually spent. If the shortlist is wrong, no reranker can fix it, so the first stage still has to be measured.

Rerank fifty, not five hundred. The quality curve flattens quickly, and the reranker is the slowest component in the query path; measure the recall of the shortlist before you make it longer.

Evaluating the combination

A hybrid system has more knobs than either retriever alone: the fusion constant, the number of candidates from each side, the reranker cutoff. Tuning them on one query is how a demo becomes a system that only works for the person who built it. Write the defaults down, change one knob at a time, and keep the query set fixed while you do it.

Keep a labelled set of queries with the passage that should be found, and measure recall at each stage. If the answer is missing from the shortlist, the reranker cannot help. If it is in the shortlist and ranked low, that is the reranker’s job. That split is the most useful diagnostic in retrieval, and it costs one extra column in the evaluation sheet.

The sheet has one row per query and one column per stage: lexical rank, dense rank, fused rank, reranked rank, and whether the target was present at all. Reading down a row tells you where the pipeline lost the answer, which is more useful than a single aggregate score. The sheet is also the artefact you hand to the next engineer.

Rerank the shortlist

Compare the first two results from the dense order with the reranked order. Then add a candidate that contains the word eligibility but has a low dense score and watch it climb.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Which query does dense retrieval handle worst?