Every AI chat product demos well. You paste a URL, it reads your site, it answers three questions you already knew the answers to, and everyone in the room nods. Nothing in that half hour tells you what the thing will do in month three, when it has 900 documents, four people editing them, and a customer acting on an answer nobody checked.
This is the list to take into the demo. It is written for the buyer, not for the vendor, and some of these are questions we had to answer badly before we answered them well.
The failure mode that matters
Everyone worries about invented answers. They are real, and they are the easy ones. An invented answer is usually obviously wrong.
The expensive failure is quieter. The assistant answers plausibly. Nobody checks it. A customer acts on it, and you find out from the customer.
Almost every question below is a way of asking the same thing: when that happens, how fast can you find the cause?
Question 1: Can it show you where the answer came from?
Not whether it looks confident. Can it name the document, and ideally the passage.
A citation is sold as a trust signal for your visitors. It is that. But it helps whoever maintains your content more.
An answer that is wrong and cites shipping-policy.pdf tells you exactly what to edit. An answer that is wrong and cites nothing leaves you rewriting prompts and hoping.
A good answer carries its sources, the sources are the ones actually used rather than everything retrieved, and you can open the passage.
Question 2: Can you see why those passages were picked?
This is the question almost nothing on the market answers, and it is the one that decides whether you can fix a bad answer yourself.
Retrieval is a search. Searches have scores. If you can read the score of each retrieved passage, a bad answer sorts itself into one of three buckets in about ten seconds.
One warning from our own logs: on real content a genuinely relevant passage often scores between 0.35 and 0.60, not the 0.80 that tutorials imply.
A similarity threshold lifted from a blog post can switch off the meaning-based half of a hybrid search. The product keeps working. It just becomes keyword search while you pay for embeddings.
Ask what the default threshold is, and what real scores look like on content like yours. If they cannot tell you, they are not looking either.
Question 3: What does it do when your content does not cover the question?
There are exactly two designs, and the vendor has chosen one whether or not they say so.
Design one. Answer from the model's general knowledge. Fluent, occasionally right, and it will eventually invent your refund window.
Design two. Say it does not know, and hand over to a person. Less impressive in a demo. Correct in production.
The second design should not be a prompt you write and hope for. Ask whether it is a setting, whether it is enforced, and whether it can be set once for the whole account instead of per assistant.
Question 4: Where can it run, and who can run it?
A widget key sits in public HTML. That is fine, as long as the key is bound to something.
Ask:
- Can I list the domains it is allowed to answer on, including staging?
- What happens when someone lifts the key and puts it on their own site?
- Can I rotate the key without redeploying?
*.example.com covering subdomains, and a flat refusal from anywhere else, is the answer you want.
A preview host is a different origin from production, so it needs its own entry. Ask how deep the wildcard goes before you assume it covers app.eu.example.com.
Question 5: Does it capture the visitor or only answer them?
An assistant that answers a pre-sales question and lets the visitor leave anonymously has cost you the lead and charged you for the privilege.
The distinction worth probing is how it asks. A form dropped into the chat panel is still a form. An assistant that asks for a name and an email in conversation, while it answers the visitor's actual question, converts differently.
If each answer is saved the moment it arrives, somebody who leaves halfway still leaves you two real fields instead of nothing.
Question 6: Does it fit the stack you already have?
Two integration questions get muddled constantly.
Content in. How your material reaches the assistant, and how it stays current when you edit it. A plugin, a crawler, an API, or a webhook.
Leads out. Where a captured lead lands. Your CRM, your mailing list, your project tracker.
Ask for both, and get the list of integrations in writing.
Question 7: Are the limits written down?
"Contact us for volume" is a limit too. It just is not written down anywhere you can read it before you sign.
Get the numbers before you commit:
- What counts as a billable answer? An abandoned question? A test from your own team?
- What happens at 100% of your plan? Do answers stop, or do you get billed at a rate you never agreed to?
- How long is analytics history kept?
- Who pays for the model calls, and can you bring your own key?
Whatever vendor you pick, ask for the equivalent in writing.
Question 8: What happens to your content?
The checklist:
- Is my content used to train anybody's model? (The answer should be no.)
- Are my stored credentials encrypted, and can anyone read them back? (Encrypted, and nobody, including you.)
- Are visitor IP addresses stored?
- Can one workspace ever reach another workspace's rows, and is that enforced in the database or in application code?
- Is there an audit log of who changed what?
Get the answers in writing, and ask for the privacy policy behind them rather than a paraphrase in an email.
The scorecard
Take this into the next demo and make them answer live rather than in a follow-up email.
- Show me an answer with its sources, and open one of the passages.
- Show me the similarity score of each passage you retrieved.
- Ask it something my content does not cover, without editing my content first.
- Put the widget key on a domain that is not on the allow list.
- Show me where a captured lead lands in my CRM.
- Show me the page where my plan's limits are written as numbers.
- Tell me what happens on the day I hit 100% of them.
A tool that can do all seven in a live call is a tool whose team has run it in production. A tool that needs to get back to you on three of them is a tool you will be debugging blind in month three.
Questions people ask
What if the vendor cannot answer these live?
A follow-up email means either they are not sure of the answer, or it is not in the product. Either way, you will spend more time debugging than you would have spent in that demo.
Can I use this with any chatbot product?
Yes. Take the eight questions and the scorecard to any demo. The product that answers all seven live is the one whose team knows what they built.
How long should the demo take?
If they know the product well, all seven scorecard items should take twenty minutes. Budget an hour if you want to dig into the reasoning behind each answer.
Start with the vendor's documentation and use cases to see how they would handle your job, then run the scorecard when you talk to them.
