Capstone: RAG end to end

The whole pipeline, in order

A document is split into chunks, each chunk is turned into a vector, the vectors go into a store, a question is embedded the same way, the nearest chunks are retrieved, and a prompt asks for an answer that cites them. Seven steps, each small enough to test alone.

The capstone builds all seven with plain Python. The vectors are hand-built concept coordinates rather than a model’s output, and the “generation” is a template rather than a language model — but every boundary is in the right place, which is what matters.

Name what each step returns: that is the contract between stages. Splitting returns chunks with ids. Embedding returns one vector per chunk. The store returns scored ids. Retrieval returns the top k. Generation returns text plus the passages it used. A change inside one stage should not change its neighbour’s shape.

Measure before you tune

The second example adds the evaluation loop: a handful of questions with known answers, a recall check, and a printed report. It is deliberately crude, and it is the difference between a demo and a system.

Once that loop exists, every change — a different chunk size, a reranker, a new embedding model — becomes a number rather than an opinion. That is the habit worth keeping from this course.

A realistic decision looks like this. Recall at four is 0.72 with 500-character chunks and 0.81 with 800, but the prompt grows by a third and latency rises. The larger chunks are better for these questions and worse for the budget, so the next experiment is overlap, not another size change. Without the loop, the same decision is an argument.

The same pipeline with a framework

retriever = vector_store.as_retriever(search_kwargs={"k": 4})

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | model
    | StrOutputParser()
)

print(chain.invoke("How long do refunds take?"))

What a production system adds

Four things, and none of them changes the shape above: real embeddings, so meaning rather than hand-written concepts drives the search; a persistent store, so the index is built once; a reranker, for precision on the shortlist; and tracing, so a bad answer can be followed back through every step.

The last one is the most valuable in practice. A trace shows the question, the retrieved chunks, the assembled prompt and the reply together, which is the only way to tell a retrieval failure from a generation failure after the fact.

A useful trace carries timings and ids. The embedding call, the search, the rerank and the generation each get a span naming the model or index it used. When a query is slow, the trace says which stage to fix; when an answer is wrong, it says whether the passage was absent, truncated or ignored.

Where the next failure will come from

As the corpus grows, chunking stops being uniform: a ten-page policy document and a one-line FAQ need different treatment. As the product grows, filters multiply and the first question becomes which collection to search, not which passage.

Neither is a reason to redesign. Both are changes inside one stage, which is exactly what the boundaries in this pipeline are for.

The first new failure is usually selection: the corpus has two documents about refunds, one current and one archived, and the search returns both. The fix is a metadata filter on status, not a better embedding. The second is duplication: the same page under three URLs fills the top four. Deduplicate by content hash before you tune anything.

Test each stage alone

Each stage can be tested with a fixed input. Retrieval takes a question and returns ids, so a test asserts that the refund question returns billing#refunds in the top two. The prompt builder returns a string, so a test asserts contiguous numbering and a respected budget. Generation takes a prompt and returns text, so a test can use a recorded prompt and check the citation markers.

A failure then has an address. If the retrieval test passes, the prompt test passes and the answer is still wrong, the fault is in generation or in how the model used the context. That is a much smaller search space than “the RAG system is broken”.