What robots.txt controls and what it absolutely does not
robots.txt is the most misunderstood file in technical SEO. It has exactly one job, yet it keeps getting used to solve problems it cannot solve, and blamed for problems it did not cause.
Here is the distinction that clears up most of the confusion: robots.txt controls crawling, not indexing. Google can index a URL it has never crawled, and it can keep crawling a URL it will never show in results. Once that clicks, everything else about the file makes sense.
What robots.txt actually does
The file gives crawl directives to compliant bots. Disallow tells a crawler not to fetch matching URLs. That is the whole feature. It does not remove pages from the index, it does not consolidate signals, and it does not protect private content, since any bot can ignore it and any human can read it to see exactly what you are trying to hide.
Blocked URLs can still get indexed if other pages link to them. Google just indexes them without content, which is where the classic 'Indexed, though blocked by robots.txt' status in Search Console comes from, along with those thin, description-less results you sometimes spot in the wild.
The noindex trap
The most common mistake found in audits: a page carries a noindex tag, and someone blocks it in robots.txt to make extra sure. This achieves the opposite. Google has to crawl a page to see the noindex. Block the crawl and the noindex becomes invisible, so the URL can sit in the index indefinitely.
If you want a page out of the index, let Google crawl it and serve noindex, either as a meta tag or an X-Robots-Tag header. Only once it has dropped out should you even consider blocking the crawl, and at that point you usually do not need to.
Stop blocking CSS and JavaScript
This one is a leftover from a decade ago. Google renders pages. If robots.txt blocks the CSS, JS, or API endpoints your templates depend on, Google renders a broken page, and everything that depends on rendering suffers with it. Check the rendered HTML in Search Console's URL Inspection tool: if the page looks empty or mangled there, blocked resources are a prime suspect.
Syntax gotchas that bite
Rules match by the most specific path, not by file order. A longer matching rule wins, and Allow beats Disallow when the match length is equal.
Disallow: / blocks everything. An empty Disallow: blocks nothing. One character of difference between a normal Tuesday and an incident channel.
The file only applies to its own host and protocol. https://example.com/robots.txt says nothing about subdomains; each subdomain needs its own file.
Wildcards are limited: * matches any sequence of characters, $ anchors the end of the URL. Disallow: /*?sessionid= is valid; full regex is not.
User-agent groups are exclusive. If Googlebot has its own group, it ignores the * group entirely, so any global rules must be repeated inside it.
Status codes matter. If robots.txt returns a 5xx error, Google initially treats the entire site as disallowed. A 404 means no restrictions at all. Flaky robots.txt hosting is a crawl outage.
When Disallow actively hurts you
Blocking URLs that redirect means Google never sees the redirect, so the old URLs keep their zombie index entries. Blocking parameterized URLs that carry canonical tags means Google cannot read those canonicals. Blocking a path that earns backlinks strands the equity, because the signals arrive at a door Google is not allowed to open.
In most audits, the right amount of Disallow is small: infinite spaces like faceted filters and internal search results, cart and checkout paths, admin routes. If your robots.txt runs three hundred lines, it is usually scar tissue from old panics, not strategy.
AI crawlers get their own rules
By now your logs include GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and friends, and robots.txt is where you set policy for them. The useful part is that the decisions are separable. Google-Extended controls whether your content feeds Google's AI training without touching search crawling at all. GPTBot governs OpenAI's training crawls. Blocking any of them does not affect your Google search rankings.
Decide deliberately. If AI answers are a discovery channel you want to appear in, blocking every AI agent is self-sabotage. If you run paywalled or licensed content, blocking trainers while allowing search crawlers is a defensible line. Either way, write explicit user-agent groups; do not assume the * group expresses your intent.
Test before you ship
Fetch the live file with curl and confirm you get a 200 and the exact content you deployed. Ten seconds, catches an embarrassing amount.
Check Search Console's robots.txt report to see which version Google actually has and when it last fetched it.
Test specific URLs against your proposed rules before deploying, not after. Screaming Frog lets you crawl with a custom robots.txt to preview the effect.
After deploying, crawl the site with 'respect robots.txt' enabled and diff the blocked URLs against what you expected to block. Surprises here are cheap; surprises in the index are not.
robots.txt is a small file with a blunt instrument inside. Use it for crawl control, use noindex for index control, use authentication for privacy, and re-read it once a quarter. Those ten minutes are cheaper than the traffic drop when someone ships Disallow: / to production.