Crawl a website

What it does

Reads a public website and turns its pages into documents. Start from one URL and we follow the links and the sitemap, extract the readable content from each page, and re-crawl on a schedule so the knowledge base tracks the site.

What it can access

Anything a visitor with no account can see, on the domain you name. Nothing behind a login, nothing on another domain unless you add it, and nothing a robots.txt disallows -- we respect it, including the crawl delay.

Our crawler identifies itself as WelcomeAIBot. If you need to allow it explicitly, that is the name.

What it needs before you start

• A publicly reachable site. A staging site behind basic auth or an IP allowlist cannot be crawled.
• Content that exists in the HTML. A page whose text is rendered by JavaScript after load looks empty to a crawler.
• A knowledge base to put the pages in.

Authentication

None. This is the only source that needs no credential, because it reads what anybody can read.

Setting it up

1. Open a knowledge base → Add source → Crawl a website.
2. Enter the starting URL.
3. Set the page limit and the schedule. Daily is the usual answer; a site that changes weekly does not need a nightly crawl.
4. Optionally narrow it with include and exclude patterns -- /blog/* to take only the blog, /tag/* to skip tag archives.
5. Save. The first crawl starts immediately.

How syncing works

Each run starts from the sitemap when there is one and from the starting URL when there is not, follows links within the allowed paths, and stops at the page limit.

Only pages whose content changed are re-indexed -- the rest are recognised by their hash and skipped, so a nightly crawl of a stable site costs very little. A page that has disappeared since the last run is removed.

What is imported

The readable article of each page: the heading structure, the paragraphs, the lists and the tables. Navigation, headers, footers, cookie banners and sidebars are stripped, because a knowledge base full of your own menu answers nothing.

The page title becomes the document title and its URL becomes the citation link. PDFs linked from crawled pages are read as well.

How updates are handled

Documents are keyed on the URL. A page that changes updates the same document; a page that 404s or disappears from the site is removed on the next run.

A URL that changes -- a slug rename, a move to a new path -- looks like a deletion and a new page, because from the outside that is exactly what it is.

What people use it for

The fastest way to start: point it at your existing help centre or documentation site and you have an assistant in minutes with nothing to migrate. Also the right answer for content in a system we have no connector for, as long as it is public.

How to check it is working

1. Sync log shows each run: pages discovered, created, updated, skipped, failed.
2. The document list shows what was actually kept, with its URL.
3. Open one document and read it. If it is full of menu text, the extraction found no main content -- narrow the crawl or use a different source.
4. Ask the Playground something only one page answers.

Common errors

"Blocked by robots.txt" -- the site disallows crawling, at least for that path. Change the site's robots.txt, or allow WelcomeAIBot specifically.

Almost nothing was found -- either the page limit was reached, or the site renders its content in the browser. View source on a page: if the text is not there, we cannot see it either.

Pages are all menu and footer -- the extraction could not find a main content region. Narrowing the include pattern to the article paths usually fixes it.

"Rate limited" or timeouts -- the site is refusing our request rate. The crawler slows itself down when this starts and speeds up again when it stops, so most of it is ridden out; if it persists, lower the schedule frequency or ask whoever runs the site to allow our crawler.

A run starts well and then most pages fail -- that is the same thing seen from the other end. Some sites answer a burst of requests with a placeholder rather than a refusal, and a page that is a couple of hundred bytes has nothing in it to index. The Sync log names the status, and a document that failed this way says how many bytes it read: a few hundred is a placeholder, tens of thousands is a real page that our extractor could not make sense of. Running the crawl again usually picks them up.

The site changed and answers are stale -- check the schedule. A crawl set to weekly is a week behind by design.

Reconnecting

There is no credential, so there is nothing to reconnect. If a crawl started failing, run it again from the source row, and read the Sync log entry -- it names the first URL that failed and why.

Disconnecting

Pause the source to stop the schedule while keeping everything it has fetched. Delete it to stop permanently; that asks whether to keep the documents, and "connection only" leaves them answering.