Meet the AI crawlers reading your site
Open a server log from any reasonably visible site and a new cast of characters shows up between the Googlebots and the Bingbots. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot. They are not search engines in the classic sense, and deciding what to do about them has quietly become part of the technical SEO job.
Not all AI bots want the same thing
It helps to sort them into three groups, because the trade-offs are different for each.
Training crawlers collect content to train future models. GPTBot (OpenAI), ClaudeBot (Anthropic), and CCBot (Common Crawl, whose corpus many labs use) are in this group. Google-Extended is not a crawler at all but a robots.txt token that tells Google not to use pages fetched by regular Googlebot for its AI training.
Search-index bots build the indexes behind AI search products. OAI-SearchBot supports ChatGPT search, and PerplexityBot builds Perplexity's index. Blocking these removes the chance of being cited there.
Live fetchers grab a page in real time when a user asks about it. ChatGPT-User and Claude-User act on behalf of a person in a conversation, closer to a browser than a crawler.
The distinction matters because one robots.txt decision can have three different consequences. Blocking a training bot limits use of content in future models. Blocking a search-index bot removes citations and referral traffic from AI search. Blocking a live fetcher stops assistants from reading the page even when a user explicitly asks.
The control panel is still robots.txt
The major players publish their user agent tokens and respect robots.txt. A policy that allows AI search visibility while opting out of training looks like this.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Whether to opt out of training is a business decision, not a technical one. Publishers with paywalled or differentiated content often block training bots. A personal or commercial site that benefits from being known by assistants often allows everything, on the logic that presence inside model knowledge is a form of distribution.
One caution before copying anyone's blocklist. Google-Extended only affects AI training and grounding. Regular Googlebot handles both Search and AI Overviews, so there is no robots.txt token that removes a site from AI Overviews while keeping classic rankings. That control simply does not exist today.
Verify before you trust the logs
User agent strings are plain text and anyone can send them. Scrapers routinely impersonate GPTBot or Googlebot to slip past filters. The legitimate operators publish their IP ranges or support reverse DNS verification, so a serious audit checks a sample of requests against those ranges before drawing conclusions about who is actually crawling.
It is also worth watching volume. Some AI crawlers have earned a reputation for enthusiastic crawl rates, and on large sites they can consume real bandwidth. Rate limiting at the CDN, or a crawl-delay directive for the bots that honor it, is a reasonable middle ground between full access and a block.
What about llms.txt
The proposed llms.txt standard suggests giving language models a curated markdown summary of a site. It is easy to add and harmless, but adoption by the major AI providers remains limited and unofficial, so treat it as an experiment rather than a requirement. The signals that demonstrably matter are the ones that always have. Clean crawlable HTML, fast responses, clear structure, and content worth quoting.
The practical takeaway is to make the decision deliberately. Check the logs, see who is visiting, decide who gets access and why, and write it down in robots.txt. Doing nothing is also a policy, just an accidental one.