How AI search picks its sources and what GEO changes

For Google's AI Overviews and AI Mode, generative engine optimization is SEO. Google's guide to optimizing for generative AI search says these features are rooted in its core ranking and quality systems and need no special files, markup or writing style. ChatGPT, Perplexity and Claude run their own crawlers and indexes, so visibility there depends on which of their bots can reach and read your pages.

That split settles most GEO questions: Google's AI features draw on the Search index you already work on, the other assistants are separate search engines with their own access rules, and advice that fits neither side is mostly guesswork.

Two retrieval paths

Google describes its AI features as retrieval-augmented generation, also called grounding. The model pulls current pages from the Search index through the normal ranking systems, then writes an answer with links to the pages that support it. It also uses query fan-out: it issues a set of related searches for the subtopics of a question, which is why a page can be cited for a question it never targets word for word. Google's page on AI features and your website sets the entry bar. A page must be indexed and eligible to show in Search with a snippet, and there are no additional requirements.

ChatGPT, Perplexity and Claude work in two steps you can see in your server logs. A search crawler builds the assistant's own index (OAI-SearchBot, PerplexityBot, Claude-SearchBot). A user-triggered fetcher loads a specific page live when a conversation calls for it (ChatGPT-User, Perplexity-User, Claude-User). Training crawlers such as GPTBot and ClaudeBot are a third and separate group. They collect material for future models, not for today's answers, so blocking them has no direct effect on citations.

The table maps each engine to its controls, based on each operator's own documentation.

Two traps sit behind that table. Blocking Googlebot to stay out of AI Overviews removes a site from Google Search altogether, so it is the wrong tool. And user-triggered fetchers do not all follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User and that ChatGPT-User plays no part in deciding what appears in ChatGPT search. Perplexity says Perplexity-User generally ignores robots.txt, and Google says the same of its own user-triggered fetchers. Anthropic says all three of its bots, Claude-User included, honor robots.txt.

Access decides the third-party side

An assistant you have never thought about still needs the basics from your site: a 200 response, the main content in the HTML, and no bot wall in front of it. Check these in order, cheapest first.

1. Read robots.txt group by group. A crawler obeys only the most specific group that names it, so a group added just to allow OAI-SearchBot also cancels every rule in the * group for that bot. The robots.txt guide covers the syntax.

2. Check what your CDN or firewall serves these user agents. Bot-management products often challenge or block AI crawlers that robots.txt allows, and the only evidence is in the status codes they receive.

3. Confirm the main content is in the server response. Google renders JavaScript before indexing, but none of the AI operators documents whether their crawlers do, so content assembled in the browser is a risk you can remove by rendering it on the server. The JavaScript SEO guide shows how to compare raw and rendered HTML.

4. Look for real visits in the logs. If OAI-SearchBot, PerplexityBot or Claude-SearchBot never request your important URLs, those pages are not in that index and no content change will get them cited. Check request IPs against each operator's published ranges first, because these user agents are trivial to fake. The log file analysis primer covers the setup.

Google's controls sit elsewhere

For Google, the choices live in four places, and only one of them is a robots.txt token.

• Googlebot access and indexing rules govern Search as a whole, AI features included.

• Snippet controls (nosnippet, data-nosnippet and max-snippet) limit what Google can show from a page, and Google lists them as the way to limit what appears in AI features. A noindex removes the page from Search entirely.

• Google-Extended has no crawler of its own. It controls whether content Google crawls can be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI, and Google's crawler documentation states that it does not affect inclusion or ranking in Google Search.

• In June 2026 Google began rolling out a Search Console setting called Search generative AI, found under Settings, to a first group of site owners. Choosing exclude keeps a site's links and content out of AI Overviews, AI Mode and the generative AI features in Discover, and Google says the setting is not used as a ranking signal for the rest of Search. It does not cover model training, which remains Google-Extended's job.

Most sites should leave all four at their defaults. Excluding a site from AI features gives up visibility in a prominent part of the results page, and the case for doing so depends on whether those impressions ever turned into visits. The measurement section below is how you find out.

Content that gets cited

Google's guidance on content for AI search separates commodity content, the common-knowledge summary anyone could produce, from non-commodity content that brings an expert or first-hand view, and says the second kind is likely to matter most over time. It also warns against publishing a page for every fan-out variation of a question. Doing that to manipulate rankings or AI responses falls under the scaled content abuse policy, and in May 2026 Google added attempts to manipulate generative AI responses to the definition of spam in its spam policies.

By that logic, what earns a citation is information the model cannot assemble from ten other pages: a comparison you measured, a procedure documented by someone who ran it, a definition precise enough to settle an argument, an original dataset.

Answer-first writing helps readers and is good editing anyway. Google says you do not need to write in a special way for its AI features or cut pages into small chunks, because its systems can find the relevant passage in a long page. For ChatGPT, Perplexity and Claude, no operator documents how passages are chosen, so claims that a particular format wins citations there are unverified.

Mentions on other sites matter, but less mechanically than GEO vendors suggest. Google says its AI features can reflect what is said about a product across blogs, videos and forums, and also that chasing inauthentic mentions is not as helpful as it might seem, because its spam systems sit underneath the AI features. Coverage you earn helps. Coverage you manufacture is a liability.

Publishers have one documented lever on the reader's side. In late May 2026 Google began extending preferred sources to AI Overviews and AI Mode, so content from a site a searcher has chosen as a preferred source can carry a preferred badge there, as it already could in Top Stories.

Agents are a newer kind of visitor

Some AI traffic reads a page in order to complete a task, such as comparing specifications or filling in a booking form. Google's guide describes browser agents that work from screenshots, the DOM and the accessibility tree, and treats preparing for them as optional work for businesses where it is relevant. Google also documents Google-Agent, a user-triggered fetcher for agents running on its infrastructure, and like its other user-triggered fetchers it generally ignores robots.txt. Preparation overlaps with accessibility work: real buttons and links, labeled form fields, and content that does not depend on hover to appear. If agents must stay out of a checkout or account area, that takes authentication or a firewall rule.

Work you can skip

Google lists several popular tactics as unnecessary for its AI features. For the other assistants the evidence is thin in both directions.

• An llms.txt file. Google says Google Search ignores llms.txt and similar AI text files, and that creating one for other services will neither help nor hurt in Google. The crawler documentation from OpenAI, Anthropic and Perplexity does not mention it.

• Special schema for AI. Google says structured data is not required for generative AI search and there is no special markup to add. Markup still matters for rich results, which the guide to schema types that still earn rich results covers type by type.

• Chunking pages into short fragments, or rewriting them to cram in every question variant. Google calls neither necessary.

• Buying or seeding mentions, for the reason above.

Measuring AI visibility

Start with what Google reports. Clicks and impressions from AI Overviews and AI Mode are counted in the Search Console Performance report under the Web search type, mixed in with everything else. In June 2026 Google began rolling out Generative AI performance reports to a subset of sites, showing impressions in AI Overviews, AI Mode and Discover's AI features by page, country, device and date. The launch post lists impressions, not clicks or queries, so read it next to the main Performance report. The brand split, annotations and data limits that make any Search Console number reliable are covered in the guide to SEO reporting.

Outside Google, three habits cover most of what can be measured.

• Create an analytics channel for AI referrals that matches referrers such as chatgpt.com, perplexity.ai, claude.ai and gemini.google.com. OpenAI's publisher documentation says ChatGPT adds utm_source=chatgpt.com to the links it sends, which makes those visits easier to separate.

• Watch search-crawler requests in verified logs. A page that OAI-SearchBot or Claude-SearchBot has never fetched cannot be in that index.

• Keep a fixed panel of the questions that matter to the business, run it monthly in each assistant, and record whether the site is cited and which URL. Answers vary between runs, users and locations, so read the panel as a trend line, never as a single score.

Google also says clicks from results pages with AI Overviews are higher quality, meaning people spend more time on the site. That is Google describing its own product. Check it against your analytics before repeating it to stakeholders.

A worked example

Take a fictional developer-docs site. It ranks on the first page of Google for most questions about its API and shows up as a link in AI Overviews, yet ChatGPT answers about its product cite a competitor's blog and a forum thread. Here is the order of checks.

1. robots.txt. The file has a group that disallows /search/ and /internal/, and a GPTBot group that disallows everything. OAI-SearchBot is not named, so it follows the group and may crawl the docs. Nothing to fix here.

2. CDN response. A request sent with the OAI-SearchBot user agent gets a 403 from the bot-management layer, which classes it as an unknown automated client. The logs show genuine OAI-SearchBot requests from OpenAI's published IP ranges getting the same 403. That is the cause.

3. The fix is an allow rule for OAI-SearchBot traffic from OpenAI's published ranges. OpenAI says its search systems can take about 24 hours to reflect a robots.txt change, and a firewall fix still needs a recrawl, so the logs are the place to confirm the bot is getting 200 responses again.

4. If citations still do not appear once the pages are fetched, the question becomes content. The competitor post that keeps getting cited has a benchmark table the docs lack. That is a gap a page can close. A new file or a new schema type cannot.

If you do one thing this month, request your three most important templates with the user agent of each search crawler and user fetcher in the table above, record the status codes, then confirm the same bots appear with 200 responses in verified logs. Access problems can be fixed in an afternoon. Everything else on this page takes months.