There are two confident answers to this question and both are wrong. One says block everything, because your content is being taken. The other says allow everything, because visibility is all that matters. Neither survives ten minutes with your own server logs.
The reason is that "AI crawler" covers three different kinds of visitor, and they do not deserve the same answer.
Three agents, three jobs
The third row is the one people block by accident. It is not a crawler harvesting your site; it is a visitor who typed your URL into an assistant. Blocking it means somebody asking about your product gets told the page could not be read.
Pick a policy and see what it costs
Each of these is a legitimate choice. What matters is knowing which trade you are making.
What we do, and why the two hosts differ
We run two hosts and they have opposite policies, which is the clearest way to explain the reasoning.
The split is the point. "Should I block AI crawlers" is rarely a question about a company; it is a question about a particular set of pages. Public marketing content and private customer data end up on opposite sides of it, and most sites have both.
User-agent: * with a Disallow covers crawlers that read the wildcard group, but several AI agents look for their own name first and follow that group instead. If you mean to block them, name them. And check what you already have: a robots.txt written years ago may be blocking a search agent nobody thought about.
What blocking actually buys you
Less than the argument suggests, and it is worth being clear about which parts are real.
- It does not unpublish you from models already trained. Anything crawled before you changed the file is already where it is.
- It is voluntary. robots.txt is a convention honoured by the agents that choose to honour it. Enforcement happens at your edge or in your terms, not in that file.
- It does not stop people. Someone can paste your page into an assistant, and an on-demand fetch is a reader, not a crawl.
- It does stop the well-behaved crawlers. Which is genuinely most of the volume, and the reason the setting is worth having.
When blocking is clearly right
Not every site should be quotable, and these cases are not close calls: content somebody pays to read, an original dataset or database that is the product itself, licensed material you do not own outright, anything users posted expecting a small audience, and any page behind a login. In each one the content is the asset, not the advertisement for the asset.
The opposite case is just as clear. If your pages exist to explain what you sell, being quoted is the job, and a policy that prevents it is an own goal dressed as caution.
welcomeai.devOur crawler reading your site, which is the other direction entirelyQuestions people ask
Decide it once, per host, and write down why. Then revisit it when you next look at your logs, because the list of agents changes faster than anybody's policy does.
welcomeai.devWhat is RAG? Retrieval-augmented generation explained
