EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

BM25
A word-matching score that ranks a document higher when it contains rare query words, with a cap on repetition.
chunk
A short passage cut from a longer document, small enough to retrieve and cite on its own.
citation
A pointer from an answer back to the exact passage it came from, so a reader can check it.
context window
The fixed amount of text a model can read in one request, counting the question and everything retrieved.
corpus
The whole collection of documents a system searches.
cosine similarity
A score from -1 to 1 for how closely two arrows point in the same direction, ignoring their lengths.
dot product
Multiply matching numbers in two lists and add the results; a larger total means closer agreement.
embedding
A list of numbers that places a piece of text in a space where similar meanings sit close together.
grounded answer
A reply built only from the passages that were retrieved, so its claims can be traced to them.
index
A structure built ahead of time so the system can find relevant passages without reading everything.
lexical search
Finding passages by the words they contain, matching the query's terms rather than its meaning.
magnitude
The length of a vector, found by squaring its numbers, adding them and taking the square root.
reranking
Scoring a short list of results again with a slower, more careful method and putting the best first.
retrieval
The step that selects passages from the corpus to put in front of the model.
top-k
The shortlist a search returns: its best few results, cut off at a number chosen in advance.
vector
An ordered list of numbers, where each position is one measured property.

RAG from the ground up

Documents and chunks

Retrieval returns chunks, not documents. How you cut the text decides what can be found, what a citation points at, and whether an answer is missing its first half.

Part 2 of 10

Beginner15 min readBack to Courses

A chunk is the unit of everything

The text you index is cut into chunks, and each chunk is embedded, searched and returned as one item. That makes the chunk the unit of retrieval, of citation and of the context the model sees. Get the cut wrong and no amount of ranking fixes it: a passage that begins mid-sentence has already lost the subject it was about.

Chunks also carry metadata — source, title, section, page. That metadata is what turns a passage into a citation, and what lets a filter narrow a search to one product or one year before ranking begins.

Chunk boundaries are also quote boundaries. If a sentence is split, the half that reaches the prompt can be quoted as if it were the whole claim, and the citation points at a fragment that reads as a different statement. Keeping sentences whole is not cosmetic; it decides whether a citation can be trusted.

The size trade-off is real

Small chunks match precisely and cite neatly, but they often lack the context needed to interpret them. Large chunks carry context but blur the embedding: a page about five topics looks vaguely like every question about any of them, and the top result is mediocre for all five.

There is no universal size. Documentation with short sections works well around a few hundred words; legal text and transcripts need more. The useful move is to measure: build the index, ask twenty real questions, and look at what comes back.

Picture a large chunk that covers pricing, refunds and support. A question about the refund window matches it weakly, because the embedding averages three topics, while a smaller chunk that is only about refunds outranks it. The large chunk is not wrong; it is diluted. The symptom is a top result that mentions the right words and answers the wrong part.

Splitting that respects structure

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=["\n## ", "\n\n", "\n", " "],
)

chunks = splitter.split_documents(documents)
print(len(chunks), chunks[0].metadata)

A window with overlap

Split the text, then change the overlap to zero and to half the window. Look at where the cuts land and at how much text is duplicated.

Output
Click "Run" to execute your code

Overlap is insurance, not a strategy

Overlapping the end of one chunk with the start of the next keeps a sentence from being split across an invisible boundary. A hundred or so tokens is usually enough; doubling the overlap doubles the index and the cost of every search.

It does not repair bad boundaries. If the split ignores headings and paragraph structure, a 40 percent overlap just produces two chunks that are both confused.

Overlap also has a cost that shows up in the results. Two chunks that share a third of their text both match the same query, so the top four can contain the same paragraph twice and one fewer distinct source. If the answers feel repetitive, or citations keep pointing at neighbouring ids, the overlap is too large for the chunk size.

Prepend the document title and section heading to every chunk before embedding it. A chunk that says “Configure the timeout” is far easier to match when it also says which product it belongs to.

Symptoms of a bad cut

Bad chunking has a small set of signatures. A chunk begins with “It also” or “This means” and the noun it refers to is in the previous chunk. A table row arrives without its header. A heading sits at the end of one chunk while its section is in the next. A policy is split across two chunks and neither half contains both the condition and the exception.

Read twenty random chunks before you tune anything. If they do not read as something a person would quote, retrieval is working against the text, and no embedding model will repair it.

Cut on meaning where you can

A heading is a statement that the text below is about one thing. Splitting on headings gives chunks that are already coherent, and the heading becomes free metadata. Splitting every 800 characters ignores that signal and cuts through the middle of explanations.

When the source has no structure — a transcript, a scanned PDF — a sliding window over sentences is the fallback. Keep sentences whole, and keep the speaker and timestamp in the metadata.

Split on headings, keep the metadata

The splitter keeps each section whole and carries its heading into the metadata. Add a second document and confirm that chunks from different documents never merge.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What is the unit that gets embedded and returned?