Documents and how they are indexed

A document is one piece of content: a file you uploaded, a page we crawled, an article a connector sent, a Q&A pair you typed.

What you can add

Four ways, all of them under Add source on the knowledge base:

• Upload files. PDF, DOCX, PPTX, XLSX, CSV, Markdown, plain text, HTML and JSON. Each file becomes a document.
• Add a web page. One URL. We fetch it once, extract the main content and index it. For a whole site, use the website crawler instead.
• Add text. Paste anything, Markdown included. Headings help the chunker keep context, so a pasted policy reads better than a pasted wall.
• Add a Q&A pair. One question and the canonical answer, indexed as its own document. This is the fastest fix when the assistant answers something badly: write the answer you wanted.

Plus anything a connector brings in, and anything you push through the REST API or the sync webhook.

A PDF that is a scan of paper has no text in it. We do not read images, so a scanned document arrives empty. Run it through OCR first -- and you will know, because a file we cannot read at all fails with "No text could be extracted from this document" rather than sitting there looking indexed.

The harder case is the one in between: a slide deck exported as images with a heading on each slide gives us a little text, indexes cleanly, and then answers almost nothing. When a document comes out with very little text for its length we mark it, and the warning beside its status says how much came out and what to do. Export it again with selectable text and re-index.

What happens after you add one

1. Extraction. We pull the text out and keep the structure -- headings, lists, tables, and which page each passage came from, so a citation can say "page 7".
2. Chunking. The text is cut into passages of a few hundred words with a little overlap, so a sentence that straddles a boundary is not lost.
3. Embedding. Each chunk is turned into a list of numbers standing for its meaning. This is what makes "can I get my money back" find a paragraph about refunds.
4. Indexed. Now it can be answered from.

The whole thing takes seconds for a page and a minute or two for a long PDF. Status tells you where it is, and a failure says why rather than sitting at "processing" forever.

Re-indexing

Changing the chunk size, the overlap or the embedding model changes how every existing document was processed, so those settings offer to re-index the knowledge base. It costs embedding tokens and a few minutes; nothing is unavailable while it runs.

Editing the text of one document re-indexes only that one.

Editing a Q&A pair

A Q&A pair is the one document you wrote yourself, so it is the one you will want to change. Its question is editable in the list, like any other title -- click it, type, press Enter. Its menu has Edit question and answer for both at once.

Both re-index the pair, and they have to: on a Q&A pair the question is not a label on the row, it is the first line of what gets indexed. That is what the assistant matches a visitor's wording against. Changing it without re-indexing would show you the new question and go on answering the old one.

Editing keeps the document. Its id, its history and its place in every answer that already cited it stay where they were, which is what deleting the pair and adding it again could never do.

Re-indexing is refused while one is already running, and for five minutes after the last one started. Every run reads and re-embeds every document, so the button says how long is left rather than quietly doing the work twice.

Letting visitors download a file

A document's menu has Visitors may download. With it on, when the assistant cites that document in an answer, the citation offers the file instead of the page it came from -- so somebody who asks about your deck can have the deck.

Three things worth knowing before you turn it on:

• It applies to anyone using that widget, not only signed-in people. That is what "visitors" means, and it is the point of the setting.
• The file is never given a permanent address. Each download is a link that works for five minutes and is checked at the moment it is used, so a link copied out of a page stops working almost at once.
• A document the assistant may not talk about to a stranger cannot be downloaded by one either. The two rules are the same rule.

Turning it off takes effect immediately.

Downloading what we hold

Every document's menu has Download, and what it gives you depends on what there is.

For a file you uploaded, it is the file: the same bytes, unchanged.

For anything a connector brought in -- a Shopify product, a Webflow item, a crawled page -- there is no original file to give you. What we store is the text we built from their record, and everything else we know about it lives beside that text: a product's stock, its vendor, its status, its handle, its image URLs, when it last changed. So the download is the whole record: a Markdown file whose front matter carries every field, followed by the content itself.

For a crawled page that content is the page as Markdown -- headings, paragraphs, lists and links -- and not the HTML it arrived as. The navigation, the footer, the inline logos and the analytics were never part of what the assistant reads, and they are not part of what you get either.

---
title: Merino crew neck
url: "https://acme.myshopify.com/products/merino-crew-neck"
source: Shopify
chunks: 3
commerce_kind: product
status: ACTIVE
vendor: Acme
total_inventory: 42
---

Our own bookkeeping is left out -- content hashes, the extraction quality score, and the note about any credential found in the text. Those belong on the screen, not in a file you are about to send to somebody.

Deleting

Deleting a document removes its chunks, its embeddings and the file we stored. It is immediate and it is not recoverable, which is why the confirmation names the document.

Deleting the source a document came from is a different question, and the product asks it: you can remove a connection and keep its documents, which is usually what you want -- they stop updating and keep answering.

Two things worth knowing

More is not better. A knowledge base with your ten clearest pages answers better than one with two hundred pages of everything, because retrieval has fewer near-misses to choose between. Old versions of a policy are worse than no policy.

Titles matter more than you would think. They are what a citation shows, and a document called "Untitled" or "export-final-2.pdf" makes an answer look untrustworthy even when it is right. The title is also searched, so a file named for what it is -- "2026 price list" rather than "doc3" -- can be found by somebody asking for it by name, even when those words appear nowhere inside it.