Why your chunk size is the answer to most bad answers
Eight hundred tokens with a hundred and twenty of overlap is a good default, not a law. Here is how to read your own retrieval scores and pick a better one.
A confident answer built on a passage that scored 0.61 is not a model failure. It is a chunking failure that arrived late.
Read the scores, not the answers
Every answer in WelcomeAI carries the passages it used and how similar each one was to the question. That number is the fastest diagnostic you have, and almost nobody looks at it.
Open a conversation where the answer was wrong. If the top passage scored above 0.80 and the answer is still wrong, the content is wrong. If the top passage scored 0.6 and the assistant answered anyway, retrieval failed and the model papered over it.
{
"chunk_size": 800,
"overlap": 120,
"threshold": 0.78,
"top_k": 6
}Those are the defaults. They are a starting point chosen because they work for help centres and product documentation, which is what most people index first.
Three symptoms and what they mean
Before touching a slider, name the failure.
- The answer is right but cites the wrong page. Your chunks are too large. One passage covers three topics, so the one that matched is not the one that answered.
- The answer misses a detail that is definitely written down. Your chunks are too small, or the overlap is too low, and a sentence got cut in half between two of them.
- The assistant says it does not know, and it should. Either the content is not indexed, or the threshold is above what your content can score. Check the document status first; it is usually that.
Picking a number
Longer documents with headings tolerate larger chunks, because a heading gives every passage under it context. A FAQ where each answer is two sentences wants smaller ones.
The overlap matters more than people expect. At 120 tokens, a sentence that lands on a boundary still appears whole in the next passage. At 20, it does not.
Change one number, re-index, and ask the same five questions you asked before. Five real questions beat any amount of reasoning about the ideal size.
When to re-index
Re-indexing rewrites every vector in the knowledge base. On a thousand documents it takes about four minutes, and the widget keeps answering from the old index until it finishes, so there is no window where your site is broken.
That is deliberate. You should feel free to try a number, look at the scores, and try another.
Builds WelcomeAI. Spends most days reading similarity scores that should have been higher.