Grounding answers with RAG, and why retrieval beats memorisation

The first time I asked a language model a question about a document it had never seen, it answered confidently and completely wrongly. That failure is the whole reason retrieval-augmented generation exists.

The problem with asking nicely

A model’s weights are frozen at training time. Anything it says about your specific PDF — a syllabus, a research paper, a contract — is reconstructed from patterns it saw elsewhere. It sounds right. It is often not right.

Retrieval flips the arrangement. Instead of hoping the answer lives in the weights, you go and fetch the relevant passage first, then ask the model to answer using only that passage.

The shape of the pipeline

Four steps, and each one can quietly ruin the output:

Extract. Pull raw text out of the PDF. PyPDF2 handles the common cases; scanned documents need OCR before they are text at all.
Chunk. Split the text into pieces small enough to retrieve precisely, but large enough to carry meaning. Overlapping windows stop a sentence from being cut in half at the seam.
Embed and store. Each chunk becomes a vector in Chroma. Similar meaning, similar direction.
Retrieve and answer. Embed the question, pull the nearest chunks, hand them to the model as context.

The interesting engineering is not in step four. It is in step two.

What I got wrong

My first chunker split on a fixed character count. It worked until a definition landed half in one chunk and half in the next — and then the retriever confidently returned the wrong half. Splitting on semantic boundaries, with overlap, fixed more of my accuracy problems than any prompt change did.

The second mistake was trusting similarity scores blindly. A chunk can be the closest match and still be irrelevant, especially when the answer is not in the document at all. Setting a floor, and letting the system say “I don’t know”, made it far more useful than letting it reach.

Where it leaves me

A hybrid setup — the uploaded knowledge base as the primary source of context, the model as the thing that phrases the answer — gets you responses that are precise because they are grounded, not because the model is clever.

That distinction turns out to matter a lot.