What robots.txt controls and when to use noindex instead

robots.txt tells crawlers which URLs they may fetch, and that is all it controls. To keep a page out of Google's results, let Google crawl it and serve noindex (or return 404 or 410, or put it behind a login), and to say you do not vouch for a link, put rel="nofollow", "sponsored" or "ugc" on that link.

Trouble starts when one of these controls is asked to do another one's job, so start from the job and then pick the tool.

Match the goal to the control

The same four directives get confused most often, so here they are side by side.

The third row explains most of the damage. noindex and canonical only work on URLs Google is allowed to fetch, and Disallow is the one control that stops the fetch.

Disallow plus noindex cancels itself out

The classic mistake goes like this. A page gets a noindex tag, and someone also blocks it in robots.txt to make doubly sure. Google's noindex documentation is blunt about the result: when robots.txt blocks the page, "the crawler will never see the noindex rule", and the URL can keep appearing in results if other pages link to it. In Search Console these show up as "Indexed, though blocked by robots.txt", and in results as a bare URL with no description.

To clear URLs stuck in that state, work in this order:

1. Remove the Disallow rule that covers them and leave the noindex in place.

2. Run a few through URL Inspection and confirm Googlebot receives the noindex, either in the HTML or in the response header.

3. Wait for recrawls. Google says a less important page can take months to be revisited, the same lag behind how long SEO changes take to show up. Request recrawls for the URLs that matter and use the Removals tool to hide urgent ones meanwhile.

4. Once they have dropped out, decide whether to block again. For a pattern that keeps producing junk, such as filter parameters, re-blocking saves crawling. For a handful of pages, crawlable plus noindex is the safer end state, because a new link to a blocked URL can put it back in the index as a bare URL.

There are two ways to serve noindex, and the header is the only option for files that have no HTML head:

<!-- in the head of an HTML page -->
<meta name="robots" content="noindex">

# as an HTTP response header, for PDFs and other non-HTML files
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

# nginx example: add the header to every PDF
location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}

Two smaller conflicts belong in the same family. A page that carries noindex and also points its canonical at another URL sends two instructions at once: drop me, and treat me as a copy of that page. Pick one. Use the canonical when the page is a duplicate whose signals should pass on, and noindex when it should not be in results at all. Likewise, an XML sitemap that lists noindexed or blocked URLs asks Google to crawl pages you have told it to ignore.

How Google reads the file

Google's robots.txt specification is short and precise. This annotated file shows the three rules that trip people up most.

# Every crawler that has no group of its own
User-agent: *
Disallow: /cart/
Disallow: /search

# Googlebot reads only this group and skips the * group,
# so the shared rules are repeated here
User-agent: Googlebot
Disallow: /cart/
Disallow: /search
Allow: /search/help
Disallow: /*?*sessionid=
Disallow: /*/print$

Sitemap: https://www.example.com/sitemap.xml

First, groups are exclusive. A crawler follows the single group whose user agent matches it most specifically and ignores every other group. Google merges several groups that name the same crawler, but it never mixes a specific group with the group. Adding a new Googlebot group without copying the shared rules into it unblocks everything that only lived under .

Second, order in the file does not matter. Google applies the rule with the longest matching path, so /search/help stays crawlable because "Allow: /search/help" is longer than "Disallow: /search". When an Allow and a Disallow match with the same length, Google picks the less restrictive one, which is the Allow.

Third, pattern matching is minimal. The wildcard matches any run of characters and $ anchors the end of the URL, so "//print$" blocks /shoes/print but not /shoes/print?page=2. There is no other regex, and paths are case-sensitive, so "Disallow: /Cart/" leaves /cart/ open.

A few limits are easy to miss:

• The file covers only the protocol, host and port it is served from. https://www.example.com/robots.txt says nothing about https://shop.example.com or the http version of either.

• Google reads the first 500 KiB and ignores anything after that.

• Google generally caches the file for up to 24 hours, longer if it cannot refresh it, so a fix can take a day to apply. The robots.txt report in Search Console shows the version Google holds and lets you request a recrawl in an emergency.

• Google supports four fields (user-agent, allow, disallow and sitemap). Crawl-delay lines are ignored, and noindex inside robots.txt is not supported.

• Google follows at least five redirect hops to reach robots.txt, then treats the file as missing.

A failing robots.txt can pause crawling

The status code of the robots.txt response matters as much as its content. Per the same specification:

Two consequences follow. A 401 or 403 on robots.txt, which can happen when a bot-protection layer challenges Google's fetch, tells Google there are no restrictions at all. And 429 is excluded from the 4xx rule, so a rate-limited robots.txt should be treated as an outage, not as an open door. Serve the file from the same reliable layer as the home page and alert on any response other than 200.

Where Disallow backfires

Blocking a URL also blocks everything Google would have learned by fetching it:

• Disallow old URLs that 301 to new ones and Google never sees the redirect, so signals stop moving to the new URLs during a migration.

• Disallow parameter URLs that carry a canonical tag and Google cannot read the canonical, so the duplicates are never consolidated.

• Google renders pages before indexing them. Blocking a resource the page does not need, such as a third-party widget script, does no harm, but blocking the CSS, JavaScript or API responses a template needs leaves Google with a broken or empty page. That is the same failure covered in this guide to JavaScript SEO.

• A blocked page that earns external links collects them at a door Google cannot open.

What belongs in Disallow is usually a short list: URL spaces with no natural end (filter combinations, sort orders, internal search results, calendars), cart and checkout flows, and session or tracking parameter variants. Admin areas need a login, not a robots.txt line that advertises where they are.

Long files tend to accumulate rules nobody can explain. Before deleting one, check whether any live URL still matches it and whether anything links to those URLs. A rule that matches nothing can go. A rule that matches URLs with traffic or links needs a decision from someone who knows why it was added.

nofollow, sponsored and ugc

These are attributes on links, so they say something about one link, not about the destination URL. Use rel="sponsored" for ads and paid placements, rel="ugc" for links inside comments and forum posts, and rel="nofollow" for other links you would rather Google not associate with your site. Values combine, so rel="ugc nofollow" is valid. A robots meta nofollow applies the same hint to every link on the page.

Google introduced sponsored and ugc in 2019 and began treating all three values as hints, and its page on qualifying outbound links says such links will generally not be followed. The destination can still be crawled if Google finds it through a sitemap or another site's link, so nofollow neither blocks crawling nor removes anything from the index.

For internal links, the same Google page says that if you do not want a URL on your own site crawled, the tool is robots.txt Disallow. nofollow on internal navigation pointing at pages you do want indexed only makes them harder to discover. Pages that should stay out of results should carry noindex instead.

Setting a policy for AI crawlers

robots.txt is also where you tell AI companies' crawlers what they may fetch, as long as the crawler honors it. The useful thing to know is that user agents now split by purpose, so training, search and Google Search can be decided separately. Each row below comes from the operator's own documentation.

Google-Extended deserves a note of its own. Google's list of common crawlers describes it as a product token with no separate HTTP user agent string. Google's existing crawlers do the fetching and read the token from robots.txt, so it will never appear in server logs, and it does not govern AI Overviews or AI Mode. Those are part of Search. They follow Googlebot's access plus the page-level controls in Google's robots meta tag documentation, where nosnippet and max-snippet also limit how much of a page can be used as direct input for AI Overviews and AI Mode. To exclude only certain passages, data-nosnippet works on parts of the HTML, as covered in this post on why Google rewrites title tags.

OpenAI's bot documentation says each of its settings is independent, so a site can allow OAI-SearchBot and disallow GPTBot. Fetchers that visit a page because a person asked an assistant to open it, such as ChatGPT-User, are a separate category. Operators treat them as user actions rather than crawling, and some say robots.txt does not apply to them, so read each operator's current page before counting on a Disallow.

A policy needs answers to a few questions first:

• Do AI answers send visitors you want? Then allow the search-oriented crawlers even if you refuse training.

• Is the content licensed, paywalled or the product itself? Refusing the training crawlers is a coherent line to hold.

• Is a bot hurting the server? That is a job for rate limiting at the firewall, matched on verified IP ranges, because anyone can fake a user agent string.

• Who owns the decision? Usually legal or the business, with SEO advising on the visibility trade-off.

Here is a file for a site that wants to appear in AI search answers but keep its content out of model training:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
Disallow: /cart/
Disallow: /search

User-agent: *
Disallow: /cart/
Disallow: /search

Notice that the search crawlers' group repeats the cart and search rules. Without them, those bots would follow their own group alone and crawl everything the * group was meant to block.

Release checklist for robots.txt changes

1. Diff the new file against the live one, and have someone who knows the URL structure read every changed line.

2. Test two URL lists against the new rules with a parser that follows Google's matching logic (Google publishes its own robots.txt parser as open source). One list must stay crawlable: home page, top categories, each page template, CSS and JavaScript bundles, API endpoints used in rendering. The other must end up blocked.

3. Check that a staging file with "Disallow: /" is not about to ship to production.

4. After deploying, fetch robots.txt on every host and protocol variant. Expect a 200 and the exact content you shipped, not an HTML error page or a redirect loop.

5. In the robots.txt report in Search Console, confirm Google has fetched the new version, and request a recrawl if the change fixes something urgent.

6. Over the following weeks, watch the Page indexing report for growth in "URL blocked by robots.txt" or "Indexed, though blocked by robots.txt" in sections you did not mean to touch.

7. Put the file under monitoring. A scheduled fetch that alerts on any content change or non-200 response catches the deploy nobody announced.