Most chatbot dashboards are built to make a bad week look good. Conversations are up, engagement is up, the satisfaction score is a comfortable four and a half. None of that tells you whether a single customer got what they came for.
The numbers that do tell you are rarely on the default view, because they are less flattering. They are also the only ones worth a meeting.
Same week, two readings
Flip the switch. The data does not change; only the question you are asking of it does.
The left-hand view is not wrong. It is just answering "is anyone using this?", which stops being the interesting question about a week after launch.
The four numbers worth reading
Resolved without a person
The share of conversations that ended without a handoff and without the same question coming back. This is the number that justifies the tool, and the one vendors most like to inflate by counting every conversation that simply stopped.
A conversation that ends is not a conversation that was resolved. Someone who gave up and phoned you also "ended the chat".
Asked the same thing twice
When a reader rephrases the question they just asked, the first answer missed. It is the cheapest failure signal you have, and it needs no survey: the transcript already contains it.
Handed to a person
Not a failure metric on its own. A healthy assistant hands off the questions it should not answer. Watch the trend and read the reasons, because a handoff rate that climbs week on week usually means a policy changed and the documentation did not.
Answered with nothing retrieved
Questions where retrieval came back empty or too weak, and the assistant either refused or should have. Every one of these is a page somebody has not written yet. It is the most actionable number on the list, and the most commonly ignored.
212 ratings out of 3,410 conversations is six percent of your readers, and not a random six percent. Read the score alongside the handoff and repeat-question rates, never instead of them.
What to read every week
Thirty minutes, the same five things, in this order.
- The ten conversations with the lowest top retrieval score
- Every question asked twice in the same session
- Handoffs grouped by reason, compared with last week
- The questions that got a refusal, to decide which ones need a page
- One conversation chosen at random, read start to finish
The last line is there because every metric above is a filter, and a filter only shows you what you already thought to look for.
welcomeai.devAI chatbot for customer support: what it answers and what it hands offQuestions people ask
If you only change one thing, stop reporting conversations and start reporting resolutions. The first number grows whenever traffic does; the second only grows when the assistant gets better.
