When your assistant gives a confident answer built on a passage that scored 0.19, while the answers you trust are built on passages around 0.55, that is a chunking failure and not a model failure. The model is doing its job. The wrong passages are reaching it.
Read the retrieval scores, not just the answers
Every answer in WelcomeAI shows you the passages it used and how similar each one was to the question. That similarity score is the fastest diagnostic you have. Almost nobody looks at it.
Open a conversation where the answer went wrong. Then compare its top score against the scores behind answers that came out right on the same knowledge base.
If the top passage scored as high as your good answers do and the answer is still wrong, your content is wrong or incomplete. If it scored well below them and the assistant answered anyway, retrieval failed and the model filled the gap instead of asking for help.
A threshold copied from a tutorial does not survive contact with real content. On real documents a genuinely relevant passage often lands between 0.35 and 0.60, not the 0.80 that examples imply. Read your own numbers before you decide which of them count as low.
The defaults start like this:
{
"chunk_size": 800,
"overlap": 120,
"threshold": 0.78,
"top_k": 6
}These work for help centres and product documentation, which is what most people index first. They are not laws.
What a chunk actually is
A chunk is a window over your document, and consecutive windows share their edges on purpose.
That shared edge is the overlap, and it matters more than people expect. At 120 tokens, a sentence that lands on a boundary still appears whole inside the next passage. At 20 tokens it does not, and neither chunk holds the whole idea.
Diagnose the failure by its pattern
Before you change anything, name what is breaking.
The third row catches more people than the other two. Check whether the document finished indexing before you touch a setting. It usually has, but the check costs ten seconds.
Change chunk size based on your content
Longer documents with clear headings tolerate larger chunks. A heading gives every passage under it context. A FAQ where each answer is two sentences needs smaller chunks.
Test one number at a time
Re-index without downtime
Re-indexing rewrites every vector in your knowledge base. On a thousand documents it takes about four minutes. The widget keeps answering from the old index until the new one finishes, so your site stays live.
That behaviour is deliberate. You should feel free to try a number, read the scores, and try another one the same afternoon.
Questions people ask
What retrieval score is good enough?
There is no single number. Score ranges depend on your embedding model and on how your documents are written, so the useful comparison is against your own baseline: what do the passages behind your correct answers score? Anything far below that is retrieval failing.
Should I always use the largest chunk size?
No. Larger chunks work when your documents have structure, such as headings and sections, that gives each passage context. Unstructured content and FAQs usually need smaller chunks so each passage stays focused on one idea.
How long does re-indexing take?
A few minutes on a typical knowledge base. Your widget continues answering from the old index until it finishes, so you can test changes without going offline.
Can I change settings without losing history?
Yes. Re-indexing only rebuilds the vectors. Your conversation history and your source documents stay the same.
Start by checking the retrieval scores on your wrong answers, name the pattern in the table above, then change one number before you touch anything else.
