An assistant that mixes up two versions of your product is not hallucinating. It is doing exactly what it was asked to do, with content that gave it no way to tell the difference.
The Series 1 reset instruction and the Series 2 reset instruction are about the same thing, in the same words, with one step different. To a system that retrieves by meaning, they are twins.
The same question, with and without the old manual
Look at the two scores in the first state. They are a hundredth apart, and no threshold you could set would keep one and drop the other. This is not a tuning problem.
Why this one is harder than a wrong answer
Most retrieval failures are visible. The assistant cites something irrelevant, somebody notices, the page gets fixed.
A version mix-up is invisible from the inside. Every passage retrieved is relevant, well written and true. The only thing wrong is which device, plan or country it was true for, and nothing in the question said. A customer who follows it does not come back to tell you; they conclude the instructions do not work.
Three fixes that work today
In the research literature the standard answer is to attach metadata at index time and filter by it at query time. That is worth knowing, and it is not a setting in this product today, so here is what actually works with what exists.
- One knowledge base per version, and a widget pointed at each. This is the strongest fix, and it maps to how your docs are already published: the Series 2 documentation site gets the assistant that answers from the Series 2 base. Plans include 1, 5 and 25 knowledge bases, so most teams have room for a base per product line or per major version.
- Put the version in the title and the first sentence. Not just in a breadcrumb or a page header, which does not travel with a passage. "Resetting a Series 2 device" as the heading, and "On the Series 2," starting the paragraph. Now the two passages stop being twins, and the model has something to distinguish them with.
- Retire what is retired. The question nobody asks is whether the old manual needs to be in there at all. If a product is discontinued and unsupported, deleting its documents from the base is one action and it removes the entire class of error.
Removing a page from your website does not remove what was already indexed. The document stays in the knowledge base until somebody deletes it there, or until a connected source syncs the removal. An assistant answering from a page that no longer exists is the most common version of this problem.
What to do when you have to keep both
Sometimes the old version is still supported and both have to answer. Three things make that survivable.
The third is underrated. "What changed in Series 2" is a page nobody writes for humans, and it is exactly the passage that lets an assistant answer a version question correctly instead of averaging two manuals.
welcomeai.devHow your assistant finds answers: chunks, top K and the thresholdHow to find out whether you have this problem
Take five instructions that changed between versions and ask the assistant each one without saying the version. Read what it cited, not only what it said.
If it cites the old page, you have content that should be retired. If it cites both and the answer hedges, you have a base that needs splitting. And if it answers correctly but never says which version it is talking about, you have a customer who is about to follow the wrong steps with complete confidence.
Questions people ask
Run the five-question test this week. It is the quickest way to find out whether your assistant has been confidently answering for a product somebody stopped selling two years ago.
welcomeai.devHow to write documentation an AI can actually answer from
