Why retrieval

Three problems a bigger model does not fix

A language model cannot answer from documents it has never seen. It cannot know a price that changed this morning. And when it does not know, it will often produce a fluent answer anyway, because that is what it was trained to do. None of those problems is solved by a larger model, because none of them is about reasoning.

Retrieval solves them by changing the question. Instead of asking the model to remember, you find the passages that contain the answer and put them in the prompt. The model then does what it is genuinely good at: reading a short text and writing about it.

A model’s knowledge stops at the end of its training. Ask about last month’s release and it answers from an older world — not because it is lying, but because that is the only world it has. Private data is the same problem: a policy in an internal wiki was never in the training set.

Grounded, and able to say so

An answer built from supplied passages is grounded: every claim can be traced to a sentence someone can check. That is what makes citations possible, and it is also what makes refusal honest — if the retrieved passages do not contain the answer, “the documentation does not say” is a correct reply rather than a failure.

The alternative, asking the model to answer from memory, gives you no way to tell the two cases apart. A confident wrong answer and a confident right one look identical.

A citation is the visible half of that. When an answer carries the passage number, a reader can open the source, read the sentence and decide for themselves whether the claim follows. The system does not have to be trusted; it has to be checkable.

The shape of every RAG system

question = "How long is the trial?"

# 1. retrieve — this is the part that decides the answer's quality
passages = retriever.invoke(question)

# 2. augment — the passages become part of the prompt
prompt = build_prompt(question, passages)

# 3. generate
answer = model.invoke(prompt)

# 4. cite
print(answer, [p.metadata["source"] for p in passages])

Retrieval is the hard part

Generation is mostly solved: give a capable model a short, relevant text and it will summarise it well. Retrieval is where RAG systems fail, and they fail quietly. The wrong passage produces a confident answer about the wrong thing, and nothing in the output says the context was bad.

That is why the rest of this course spends most of its time on chunking, embeddings and ranking. The prompt is four lines; the retrieval pipeline is the system.

The failure is quiet because the output is fluent either way. A passage about the team plan, retrieved for a question about refunds, produces a well-written paragraph about billing, and nothing in the text says it answered a different question.

What retrieval actually does

Under the prompt, retrieval is a search. The question is embedded into the same space as the chunks, every chunk is scored by similarity, and the best few are returned in order. Chunking decides what can be found, the embedding model decides what counts as similar, and the store decides how many candidates are considered. A prompt cannot repair a miss in any of those three.

The common mistake is to start at the prompt. If the right passage never reached the model, changing the wording only makes the wrong answer sound better. Read the retrieved chunks first; they tell you which stage to fix.

When not to use RAG

If the corpus is small enough to fit in the context window and never changes, put it in the prompt. If the task is reasoning over a database, a query is more reliable than a search. If the answer must be exact — a balance, a count — call the system that owns the number.

RAG is the right shape when the knowledge is large, changes, and needs a source. Choosing it for anything else adds a pipeline that can be wrong without adding an answer that can be right.

There is also a cost argument. Every query embeds, searches and sends several passages to the model, so a RAG call is slower and more expensive than one completion. If a lookup table can answer exactly, that call is cheaper and more correct.