Most knowledge bases get worse as they grow. The first ten documents answer well, and the hundredth makes the assistant vaguer instead of smarter.
That is not a scale problem. It is what happens when you index everything you own instead of what people actually ask.
Start from the questions, not the documents
The instinct is to point a crawler at the help centre and let it run. It is the wrong first move, because it decides what the assistant knows before anyone has decided what it needs to know.
Start with the twenty questions your team answers most often. Support inboxes, site search logs and the first five minutes of sales calls will give you all twenty in an afternoon.
Next to each question, write the document that answers it. Three kinds of row appear: a question with a good document behind it, a question with a document that half answers it, and a question nobody has ever written down.
That third kind is the most valuable thing you will produce this week.
A knowledge base cannot answer from a document that does not exist. Writing four short pages to close four gaps beats indexing four hundred pages that were never about the question.
What to index first, and what to leave for later
Documents are not worth the same to a retrieval system. Some were written as answers already. Others were written to persuade, or to be read with a person talking over them.
Six good documents will answer more questions than four hundred mediocre ones. Start narrow enough that you can read every answer the assistant gives.
Read your coverage before you index anything
The list of questions and sources is a map. Read it before you upload, because it tells you where the work is.
Green rows are ready to index. Amber rows need the document extended before it is worth indexing. Red rows are writing work, and no retrieval setting will substitute for them.
What breaks, and what it looks like when it does
Four failures cover almost everything that goes wrong in the first month.
The first row is the one teams argue about. The assistant is not being inconsistent; your documentation is, and until now nobody had a tool that read all of it at once.
Decide what does not go in
An assistant answers from whatever you feed it, to whoever can reach the widget. That is the whole value, and it is also the whole risk.
Anything indexed behind a widget on your site is effectively published. If it would not survive being on a page with no login, it does not belong in that knowledge base.
Salary bands, roadmaps, customer names and incident write-ups all read like ordinary documents to a crawler. The usual answer is two knowledge bases: one for the site, one for staff, with different widgets and different access.
Test it before your customers do
Your twenty questions are now a test set, which is the point of having written them down.
Ask all twenty and mark each answer right, wrong or partial. A first honest score is often around 14 / 20, and the six failures are more useful than the fourteen successes because each one names a specific cause.
Read the passages behind the wrong answers, not just the answers. A wrong answer built on a good passage means the document is wrong; a wrong answer built on a weak passage means retrieval missed.
welcomeai.devHow to fix bad retrieval scores by changing chunk sizeKeep it from rotting
A knowledge base is not a project that finishes. The documents behind it change, and an index that does not follow them starts answering from last quarter.
- Connect the sources that change often, so edits arrive without an upload
- Re-run the twenty questions after any large content change
- Read the questions people asked that got no good answer, monthly
- Give every source an owner, because a document with no owner is a document nobody updates
The fourth one is the least technical and the most load-bearing.
Questions people ask
Write your twenty questions first, mark the ones nothing answers, and index only what is left. The gaps you find in that hour will do more for the assistant than any setting on the page.
