Noindex, disallow, and nofollow do different jobs
Three directives, three different machines. Robots.txt disallow talks to the crawler. Noindex talks to the index. Nofollow talks to the link graph. They get mixed up constantly because they all sound like ways to say go away, and the mix-ups produce some of the most common self-inflicted wounds in technical SEO.
Disallow controls crawling, not indexing
A disallow rule in robots.txt tells crawlers not to fetch matching URLs. It says nothing about indexing, and Google can index a URL it has never crawled if enough links point at it. Those results show up with the URL and a note that no description is available, which looks broken and defeats the point. Robots.txt is a traffic-control tool for managing crawl activity, not a privacy tool and not a deindexing tool.
It is also worth remembering what disallow is good at: keeping crawlers out of infinite spaces like faceted URLs, internal search results, and endless calendars, where the goal is protecting crawl capacity rather than hiding content.
Noindex controls indexing, and needs to be crawlable
A noindex directive, in the robots meta tag or the X-Robots-Tag response header, tells search engines not to include the page in results. The header variant is the tool for PDFs and other non-HTML files. Noindex works reliably, with one catch that ties the two directives together: the crawler has to fetch the page to see it.
That produces the classic backfire. A page is set to noindex, then robots.txt blocks it, on the theory that more blocking is more blocked. Now the crawler cannot see the noindex, the page stays or becomes indexed from links, and the two directives have cancelled each other out. Rule of thumb: pick the tool that matches the goal, and never combine disallow with noindex on the same URL and expect both to work.
A related trap is leaving noindexed pages in the sitemap while blocking them from crawling, or noindexing pages that canonical to something else. Directives that argue with each other get resolved by the search engine, and not always the way anyone intended.
Nofollow is about links, and it is only a hint
The rel=nofollow attribute annotates an individual link, telling search engines not to treat it as an endorsement. It does not deindex the destination and does not block crawling on its own. Since 2019 it has company: rel=sponsored for paid links and rel=ugc for user-generated content. Google also announced then that these attributes became hints rather than commands, so a nofollowed link may still be crawled or counted in some form.
The old tactic of sculpting PageRank by nofollowing internal links deserves its retirement. Internally, nofollow mostly just tells the search engine which of the site's own pages it does not trust, which is a strange message to send. Internal links should be follow; pages that should not be indexed should say noindex.
Matching tool to job
Keep crawlers out of infinite or wasteful URL spaces: robots.txt disallow.
Keep a page out of search results: noindex, and let it be crawled so the directive is seen.
Distance the site from a link it does not vouch for: nofollow, sponsored, or ugc.
Get something removed fast: the removals tool in Search Console for the short term, plus a real noindex, 404, or authentication for the long term.
Actually protect private content: authentication. None of the three directives is a lock; they are polite signs.
None of this is exotic knowledge, yet audits keep finding disallowed pages that were meant to be deindexed and noindex tags no crawler is allowed to read. Getting the vocabulary right is a ten-minute lesson that prevents months of quiet damage.