Better retrieval
Vectors miss exact terms; keywords miss synonyms. Combine them, then reorder the shortlist with a model that reads the query and the passage together.
Two ways to fail
A vector search for “ERR_CONNECTION_RESET” is a guess: the token is rare, and its embedding sits near other network errors rather than at the exact string. A keyword search for “how do I get my money back” finds nothing, because the document says “refund”.
Each method is strong where the other is weak. Vectors capture meaning and tolerate paraphrase; lexical search captures exact tokens, codes, names and numbers. A system that uses only one has a predictable blind spot.
The blind spot is predictable enough to test for. Take twenty real questions and split them into two piles: ones that hinge on an exact string and ones that hinge on a paraphrase. A vector-only system will fail the first pile; a keyword-only system will fail the second. The failures are not random, which means the fix is not a better prompt — it is the other retriever.
BM25 is still the baseline
BM25 scores a document by the query terms it contains, weighted by how rare each term is across the corpus, and normalised so a long document does not win by length alone. It is decades old, needs no training, and is hard to beat on exact-match queries.
Its weakness is vocabulary: it only matches words that appear. That is why it is paired with vectors rather than replaced by them.
The two weights inside BM25 are worth knowing by name. Term frequency saturates: the tenth mention of “refund” adds much less than the second, because a document that says it ten times is not ten times more relevant. Length normalisation divides by document length, so a long page does not win just by containing more words. Both are why BM25 beats a raw word count.
Hybrid retrieval with a reranker
from langchain.retrievers import EnsembleRetriever, ContextualCompressionRetriever
from langchain_cohere import CohereRerank
hybrid = EnsembleRetriever(
retrievers=[bm25_retriever, vector_retriever],
weights=[0.4, 0.6],
)
reranked = ContextualCompressionRetriever(
base_compressor=CohereRerank(model="rerank-english-v3.0", top_n=4),
base_retriever=hybrid,
)
docs = reranked.invoke("ERR_CONNECTION_RESET on upload")BM25 in twenty lines
Score the documents, then add a query term that appears in every document and watch its contribution fall — that is the inverse document frequency at work.
Fusion without a shared score
A vector score and a BM25 score are not comparable — one is a cosine, the other an unbounded sum. Adding them, even with weights, is arbitrary. Reciprocal rank fusion avoids the problem by using only positions: each document scores 1 divided by a constant plus its rank in each list.
It sounds too simple to work, and it works well. What matters is that a document found by both methods rises, and a document found by one still appears.
A concrete comparison shows the effect. With the usual constant of 60, a document ranked first by one retriever and fourth by the other scores about 0.0320; a document ranked second by both scores about 0.0323 and wins. Consensus beats a single strong vote, which is the behaviour you want when the two methods fail in different ways.
Query rewriting, the cheapest lever
Before adding another retriever, try rewriting the query. A short follow-up such as “and the team plan?” is missing the subject, and no retriever can find what the query never said. A rewriting step that expands it to “how much does the team plan cost” costs one cheap model call and often moves recall more than a reranker.
The same step handles vocabulary mismatch. Ask the model for three paraphrases, search all of them, and fuse the results. It is the same reciprocal rank fusion as above, applied to one retriever instead of two.
Retrieve wide, rerank narrow. Ask each retriever for twenty candidates, fuse them, then let the reranker choose the four that reach the prompt. Reranking is the expensive step, so it should see a short list.
Reranking reads both sides
A bi-encoder embeds the query and the document separately, which is what makes searching millions of vectors fast. A cross-encoder reads the pair together and scores relevance directly. It is far more accurate and far too slow to run over a corpus.
That is the whole architecture: cheap recall over everything, expensive precision over the shortlist. Skipping the second stage is how a system returns four passages that are all about the right topic and none of which answers the question.
The cost is per pair, not per query, so the shortlist length is the budget. Reranking twenty candidates is twenty forward passes; reranking a thousand is a thousand, and the latency shows. That is why the retriever asks for twenty and the reranker returns four, rather than the reranker scanning the corpus.
Fuse two rankings by position
The two retrievers disagree. Fuse them with reciprocal rank fusion, then add a third list and confirm the documents found by several lists rise.
Check what you learned
One question at a time. A wrong answer explains itself — take another go.
Part 1 of 5
Which query is a vector search most likely to miss?