How your assistant finds answers

Your assistant does not know your business. It reads a few passages from your content every time someone asks a question, and answers from those. This page explains how it picks them, and what each setting on the Retrieval tab changes.

No prior knowledge assumed. If you have never heard of an embedding, start at the top.

The short version

1. You add a document.
2. We split it into chunks — passages a few paragraphs long.
3. Each chunk is turned into an embedding: a long list of numbers that stands for its meaning.
4. Someone asks a question. It becomes an embedding too.
5. We find the chunks whose numbers are closest to the question's. That is retrieval.
6. The best few are handed to the model along with the question. It answers from them, and cites them.

Everything on the Retrieval tab adjusts one of those steps.

Documents and chunks

A document is one thing you added: a PDF, a page we crawled, a support article, a question and answer pair.

Documents are not what gets searched. A forty-page handbook is about forty things, and returning the whole handbook because one paragraph matched would bury the answer. So each document is split into chunks.

A chunk is the unit of everything that follows. It is what gets searched, what gets retrieved, what the model reads, and what a citation points at.

Chunk size

How much text goes in each one, measured in tokens — roughly three quarters of a word, so 800 tokens is about 600 words, a page or so.

• Larger chunks carry more of their surroundings. An answer sees more context, and also more text that had nothing to do with the question.
• Smaller chunks match more precisely. Too small and a passage loses the context that made it make sense: a line reading "This is included in every plan" is useless without knowing what this is.

800 is a good default. Lower it if your content is short entries — FAQs, product records — and answers feel padded.

Chunk overlap

Chunks have to be split somewhere, and a sentence can fall right across the boundary. It then belongs to neither chunk properly, and neither one answers the question it half contains.

Overlap repeats the last part of each chunk at the start of the next, so anything near a boundary appears whole in at least one of them.

The cost is duplication: text that appears twice is stored twice and, when both chunks are retrieved, read twice. 120 tokens is a good default.

Embeddings, and why they exist

Keyword search fails on paraphrase. Someone asking "can I get my money back?" finds nothing in a document that says "refunds are available within 14 days", because they share no words.

An embedding solves this. It is a list of numbers — 1,536 of them — produced by a model that has read a great deal of text, arranged so that passages meaning similar things get similar numbers. "Refunds are available" and "can I get my money back" end up close together even with no words in common.

We store one embedding per chunk in a vector database — here, PostgreSQL with an extension that can search these lists quickly. When a question arrives it is embedded the same way, and finding the closest chunks is a matter of measuring distance. That is similarity search.

Both sides have to be measured the same way, which is why changing the embedding model re-indexes everything. Numbers from one model mean nothing to another.

Retrieval

Retrieval is that search, plus the decisions about what to keep.

Top K

How many chunks the assistant reads before answering.

This is the setting that matters most, for two reasons. It decides how likely the right passage is among them — and it is most of what an answer costs, because every chunk is text the model reads on every single question.

• Higher means more chance the answer is in there, and more chance of noise. Above about eight, weak matches usually hurt: the model has to work out which of nine passages to believe.
• Lower is cheaper and faster on every question, and often better.

Six is a good default. If answers cite things that are not relevant, lower it before you change anything else.

Similarity threshold

Similarity search always returns something. Ask about the weather and it will still hand back your three least-unrelated chunks, because they are the closest ones that exist.

The threshold is the floor: chunks that are not close enough are dropped, even if they were the best available. This is what lets your assistant say "I don't have that" instead of inventing an answer from whatever was nearest.

• Higher means fewer, better chunks — and, too high, real matches thrown away.
• Lower lets more through, including passages that only look related.

0.78 is a good default. If your assistant answers questions it should have refused, raise it.

Hybrid search

Meaning-based search is bad at exact strings. A part number, an error code, a person's name — these have no meaning to be close to, they just have to match.

Hybrid mode runs both: the meaning-based search and an old-fashioned keyword search, then merges the two rankings. It is the default and almost always the right choice.

Keyword search is also why the text search language setting exists: it is what makes "running" find "run". Set it to the language your content is written in.

Page context boost

The widget tells us which page the visitor is on. Someone asking "how much is this?" on a pricing page means that page.

This setting decides how much of a nudge chunks from the current page get. It changes nothing about cost or speed — only which chunks win a close contest.

Context, and what the model actually sees

The chunks that survive all of that become the context: the text handed to the model alongside the question and your instructions. The model answers from it.

This is why an assistant built this way can cite its sources. Each claim comes from a chunk, and each chunk came from a document — so a citation is just a pointer back to where the passage came from. It is also why the assistant does not know things you have not given it.

Why something irrelevant sometimes comes back

Three reasons, in the order they usually apply:

1. Top K is a number, not a judgement. If you ask for six chunks, you get six, even when only two are relevant. The rest are the next closest things.
2. Similarity is not correctness. Two passages can be about the same subject where one answers the question and the other does not. The numbers cannot tell them apart.
3. A chunk can lose its context. A passage split away from its heading may read as general when it was specific.

The fixes, in order: lower Top K, then raise the threshold, then look at chunk size. If a particular question keeps going wrong, adding a short question-and-answer entry that answers it directly is more effective than any amount of tuning.

How this affects what you pay

Retrieval is the largest lever you have over AI usage, and the least obvious.

Every chunk retrieved is input the model reads on every question. Going from eight chunks to five cuts roughly a third of the input on every single one — and usually improves the answers, because the three you dropped were the weakest matches.

Indexing, by contrast, is cheap: making embeddings costs about a hundredth of what answering does, and happens once per document.

So if your usage is higher than you expected:

1. Lower Top K. Biggest effect, usually improves quality.
2. Raise the similarity threshold. Drops weak matches before they are paid for.
3. Then consider the model.

There is more on how usage is measured in AI usage and limits.

When to change nothing

The defaults are good for most content. Change one setting at a time, ask the same handful of questions in the Playground before and after, and keep the change only if the answers are better.

Changing chunk size, chunk overlap or the embedding model re-indexes the whole knowledge base. Top K, the threshold, the search mode and the page boost apply immediately to the next question, with nothing to rebuild — so those are the ones to experiment with.