A hallucination is not the model lying to you. It is the model doing exactly what it was built to do — produce the most plausible continuation — in a place where nobody gave it anything to work from.
That reframing matters, because it moves the fix upstream. Almost every hallucination in a retrieval system is a gap in what was retrieved, not a defect in the model that wrote the sentence.
The three ways it happens
Name which one you are looking at before you change anything. They have different fixes and the wrong fix makes the others worse.
The third one is not a hallucination at all, and teams spend weeks tuning retrieval before they notice. The assistant is being perfectly consistent with a source that disagrees with another source.
What a good answer looks like when there is no answer
The most useful thing an assistant does is refuse. Try the third question below — it is deliberately outside the indexed material.
The refusal names its best match and its score. That is not decoration: it tells you the question was asked, the retrieval ran, and the result was too weak to build on. Now you know a page is missing.
An assistant that answers everything is not more capable. It is less honest, and you will find out which on a question that matters.
Grounding is not a guarantee
Retrieval removes the most common cause and does not remove all of them. A model handed a passage that half covers the question will still tend to close the gap rather than stop.
That is why the threshold and the instruction both matter. The threshold decides what reaches the model; the instruction decides what it does when what reached it is thin.
A customer who is told the wrong return window acts on it. The cost is not a bad conversation, it is a refund you now have to honour or a complaint you now have to lose. Silence is recoverable; a confident mistake is not.
What actually reduces it
In the order worth doing, cheapest first.
- Read the passages behind your wrong answers. A wrong answer built on a strong passage means the document is wrong. Built on a weak one, retrieval missed. You cannot pick a fix without knowing which.
- Raise the threshold until it starts refusing. Then back off one step. Most setups are tuned to answer too often because a refusal feels like a failure in a demo.
- Cut retrieval depth. Fewer, sharper passages beat more passages where half are marginal — and this reduces cost at the same time.
- Write the missing page. If a question keeps arriving and nothing answers it, no setting will conjure the content.
- Fix contradictions. Two live versions of a policy is not something an assistant can resolve on your behalf.
Questions people ask
Start by opening three answers you know are wrong and reading the passages behind them. Whatever you were about to change, that hour will tell you whether it was the right thing to change.
