How to fix bad retrieval scores by changing chunk size

When your assistant gives a confident answer built on a passage that scored 0.61, that is a chunking failure, not a model failure. The model is doing its job. The wrong passages are reaching it.

Read the retrieval scores, not just the answers

Every answer in WelcomeAI shows you the passages it used and how similar each one was to the question. That similarity score is the fastest diagnostic you have. Almost nobody looks at it.

Open a conversation where the answer went wrong. If the top passage scored above 0.80 and the answer is still wrong, your content is wrong or incomplete. If the top passage scored 0.6 and the assistant answered anyway, retrieval failed. The model filled in gaps instead of asking for help.

The defaults start like this:

{
  "chunk_size": 800,
  "overlap": 120,
  "threshold": 0.78,
  "top_k": 6
}

These work for help centres and product documentation, which is what most people index first. They are not laws.

Diagnose the failure by its pattern

Before you change anything, name what is breaking.

The answer is right but cites the wrong page. Your chunks are too large. One passage covers three topics. The one that matched the question is not the one that answered it.

The answer misses a detail that is written down. Your chunks are too small, or the overlap is too low. A sentence got cut in half between two chunks, so neither one contains the whole idea.

The assistant says it does not know, and it should. The content may not be indexed, or your threshold is too high for your documents. Check whether the document finished indexing first. It usually has.

Change chunk size based on your content

Longer documents with clear headings tolerate larger chunks. A heading gives every passage under it context. A FAQ where each answer is two sentences needs smaller chunks.

Overlap matters more than people expect. At 120 tokens, a sentence that lands on a boundary still appears whole in the next passage. At 20 tokens, it does not.

To test a new number: change one setting, re-index, and ask the same five questions you asked before. Five real questions beat any amount of reasoning about the ideal size.

Re-index without downtime

Re-indexing rewrites every vector in your knowledge base. On a thousand documents it takes about four minutes. The widget keeps answering from the old index until the new one finishes, so your site stays live.

That is deliberate. You should feel free to try a number, check the scores, and try another. The cost of testing is low.

Questions people ask

What retrieval score is good enough?

Above 0.80 means the passage is strongly relevant. Between 0.70 and 0.80 is borderline; the model may or may not find the right answer. Below 0.70 means retrieval is failing.

Should I always use the largest chunk size?

No. Larger chunks work when your documents have structure (headings, sections) that gives context. Unstructured content or FAQs often need smaller chunks so each passage stays focused.

How long does re-indexing take?

On a typical knowledge base, re-indexing takes a few minutes. Your widget continues answering from the old index until it finishes, so you can test changes without going offline.

Can I change settings without losing history?

Yes. Re-indexing only rebuilds the vectors. Your conversation history and source documents stay the same.

Start by checking the retrieval scores on your wrong answers, diagnose the pattern, and test one new chunk size before you change anything else.