Crawl a website

The crawler is the fastest way to give an assistant everything you have already published.

Starting a crawl

In a knowledge base, add a Website source and give it one or more starting URLs. We read robots.txt, look for a sitemap, and then follow links from your starting pages.

Two settings decide how far it goes:

• Maximum depth. How many links away from a starting page we are willing to travel. Three is usually right for a marketing site.
• Maximum pages. A ceiling, so a calendar or a paginated archive cannot consume your plan. Your plan sets the hard limit; you can set a lower one.

Include and exclude

Patterns are matched against the path, and ** matches anything:

• /docs/** — only the documentation.
• **/tag/** — skip tag archives.
• **?* — skip anything with a query string.

Excludes win over includes. If you crawl a large site, start with an include pattern rather than a high page limit: precision beats volume for answer quality.

What we skip

• Anything robots.txt disallows for our user agent.
• Pages with noindex, and links marked nofollow.
• Anything outside the starting site, unless you add it as another starting URL.
• Private and local addresses, always.

Re-crawling

Choose daily, weekly, or manual. A re-crawl sends a conditional request for each page, so a page that has not changed costs one small request and is not re-indexed. Pages that have disappeared are removed from the knowledge base.

Checking a crawl

The source card shows the last run and the sync log lists what was created, updated, skipped and removed, page by page, with the reason. A page that failed shows the status we got, which usually points straight at the problem: a redirect loop, a login wall, or a page that is entirely JavaScript with no server-rendered text.