A language model does not know your refund policy. It was trained on public text months before anyone asked, and your documentation was not in it.
RAG is how you close that gap without retraining anything. The model reads your documents at the moment of the question, then answers from what it read.
The four steps, every time
Retrieval-augmented generation is two words describing two halves. Retrieval finds the relevant passages; generation writes the answer using them. Nothing is memorised in between.
Step through one question below and watch what the model has, and has not, been given at each stage.
The fourth step is the one that surprises people. The model is not recalling your policy; it is reading a passage of it that arrived half a second ago, the way you would read a page someone slid across the table.
Why the documents are cut up first
Before any of this, your documents are split into passages and each passage is turned into a vector. That vector is a position in a space where things that mean similar things sit near each other.
This is why a question phrased nothing like your documentation still finds it. "Can I send this back?" and "Returns are accepted within 30 days" share no useful words, and they land close together anyway.
Why not just paste everything into the prompt
The obvious alternative is to skip retrieval and hand the model your whole help centre. Three things go wrong.
The third is the counterintuitive one. More context is not more accuracy. Retrieval is a filter, and the filter is what makes the answer specific.
What RAG is not
Three things get confused with it constantly, and each changes something different.
Fine-tuning a model on your help centre is the expensive way to get a worse result: the facts get baked in, so updating one means training again, and the model cannot tell you which page an answer came from.
Citations come free, and that is the point
Because the passages are known before the answer is written, the system can show them. That turns an answer from a claim into something checkable.
It also gives you a repair path. When an answer is wrong you can read the passage behind it and tell immediately whether the document is wrong or the retrieval missed.
Where it breaks
RAG fails at retrieval far more often than at generation. If the right passage never reaches the model, no amount of prompting saves the answer.
The usual causes are passages cut at the wrong size, documents that contradict each other, and content that was never written down. All three are fixable, and none of them are model problems.
welcomeai.devHow to fix bad retrieval scores by changing chunk sizeWhat a working setup needs
- Somewhere to keep the documents, and a way to keep them current
- An embedding model, used identically for documents and questions
- A vector index that can be searched fast enough to sit in a request
- A retrieval step that returns passages with scores you can read
- A model instructed to answer from those passages, and to say so when they do not cover the question
The fifth line is what separates a system that says "I don't know" from one that invents an answer.
Questions people ask
If you are deciding whether RAG fits, the question is not technical. It is whether the answers you need depend on documents that change, and whether the person reading them needs to see where each one came from.
