How to tell whether your site has a crawl budget problem
Most sites do not have a crawl budget problem. Google's guide to crawl budget is written for sites with roughly a million or more pages that change weekly, 10,000 or more pages that change daily, or a large share of URLs stuck in "Discovered - currently not indexed", and if none of that describes your site, missing pages are almost always a quality, duplication or linking issue rather than a crawling one.
If one of those cases does fit, the work is to find out which limit you are hitting and pull the lever that matches it.
Google's three cases
Google's crawl budget guide names three kinds of site and says the numbers are rough estimates, not thresholds. It also says that if new pages tend to get crawled the same day they are published, you do not need the guide at all.
The first two cases are about freshness at scale. A marketplace, classifieds site or large catalog with a million pages that change every week needs Google to keep revisiting them, and so does a jobs board or news site whose 10,000 pages change daily. On those sites, a stale copy in Google's index has a direct cost: expired listings in results, new ones missing.
The third case applies at any size, and it is the most direct crawl budget symptom Search Console offers. Google's Page indexing report help defines "Discovered - currently not indexed" as a URL Google found but has not crawled yet, typically because crawling it was expected to overload the site, so the crawl was rescheduled. A large and growing pile of those means Google knows about more URLs than it is willing to fetch from your server.
Keep that separate from "Crawled - currently not indexed". There Google did fetch the page and decided not to index it, which points at the page itself, and this guide to crawled but not indexed pages covers that problem.
Two limits decide how much Google crawls
Crawl budget is where two separate limits meet.
Crawl capacity is how much fetching your server can take. Google raises it while responses stay fast and stable, and lowers it when the site slows down or returns server errors (5xx) or rate-limiting responses (429).
Crawl demand is how much Google wants to fetch. The guide lists three inputs. Perceived inventory is every URL Google knows about on your site, including the duplicates and junk, and Google calls it the factor you can control most. Popularity means URLs that are more popular on the web get crawled more often. Staleness means Google recrawls often enough to catch changes. Site-wide events such as a domain move also raise demand for a while, so a site migration needs server headroom for the recrawl that follows.
The fixes for the two limits are different, so diagnose before changing anything:
| What you see | Likely limit | Where to confirm | First move |
|---|---|---|---|
| Crawl requests fall while average response time rises | Capacity | Crawl Stats response time and host status | Fix server latency before anything else |
| Bursts of 5xx or 429 answers to Googlebot | Capacity | Crawl Stats by response, then server logs | Find the layer returning errors (origin, CDN, firewall, rate limiter) |
| Steady crawl volume, much of it on sort, filter or duplicate URLs | Inventory waste | Server logs grouped by URL pattern | Consolidate or block those patterns |
| New pages wait a long time for a first crawl while old junk gets refreshed | Inventory waste, weak links to new pages | Logs, time from publish to first Googlebot hit | Remove junk, link new pages from often-crawled hubs |
| Growing "Discovered - currently not indexed" | Inventory beyond capacity | Page indexing report, sampled by URL pattern | Cut inventory first, then add capacity |
| Low crawling with a fast server and clean URLs | Low demand | The pages themselves | Work on content and links, not crawl settings |
A worked example with a hypothetical shop
Say a shop sells 20,000 products across 400 categories. Each category listing runs to about 10 pages, offers 6 sort orders and 3 page-size options, and has two single-select filters, color (10 values) and brand (25 values), which can be combined. Nothing is blocked and every option is a crawlable link.
| URL type | Count |
|---|---|
| Product pages | 20,000 |
| Default category listings, all pages | 4,000 |
| The same listings in other sort orders and page sizes | 68,000 |
| Filtered listings (285 combinations per category, about 2 pages each, every sort and size) | about 4.1 million |
The products the shop exists to sell are about half a percent of the URLs a crawler can find. Here is where each lever applies:
Sort order and page size never change which products exist, so their parameters go into robots.txt. That alone removes 17 out of every 18 listing variants.
Of the 285 filter combinations per category, a few deserve indexable URLs because people search for them (brand within category is the usual candidate). The rest are blocked or moved into URL fragments, and combinations with no products return 404. The decisions behind that are in this post on faceted navigation and pagination.
Product pages get conditional request support (more below), discontinued products return 404 or 410, and new products are linked from the first page of their category so they are found quickly.
The XML sitemaps list products and default listings only, with lastmod values that change only when the page does.
After that, the crawlable space is roughly 20,000 products, 4,000 listing pages and a few hundred promoted filter pages.
One caveat from Google's guide changes how to read the result. Blocking URLs in robots.txt does not hand the freed crawling to other pages unless Google was already hitting your capacity limit. If this shop has a growing "Discovered" pile, the block lets Google reach products it was putting off. If it does not, the block mostly saves server load and keeps junk out of Google's systems. Either way, use robots.txt for patterns you never want crawled, not as a switch to flip for a few weeks.
Levers that change crawling
These come roughly in order of how much they move on a large site.
Shrink the URL inventory
Consolidate duplicates with canonicals or redirects, block URL patterns that never deserve a crawl, and stop linking to parameter variants in templates. The robots.txt guide covers the syntax and the traps. noindex is the wrong tool here. Google's guide says not to use it for crawl control, since Google still requests the page before dropping it, and Google's myths and facts about crawling page adds that removing URLs from the index can only free crawling indirectly, over time.
Return honest status codes
A 404 or 410 is a strong signal not to crawl a URL again, and the myths page confirms that 4xx responses other than 429 do not waste crawl budget. Soft 404s, pages that return 200 with nothing on them, keep getting crawled. Long redirect chains cost a fetch per hop, so point internal links at final URLs.
Make each fetch cheaper
Faster server responses raise the capacity limit. Conditional requests go further for pages whose content stays the same between crawls. In its post on HTTP caching, Google says its crawlers support ETag and Last-Modified validators, recommends ETag, and notes that large sites whose pages seldom change benefit most. The exchange looks like this:
# First crawl: the server sends validators
HTTP/1.1 200 OK
ETag: "p-48213-v7"
Last-Modified: Tue, 02 Jun 2026 08:00:00 GMT
# Next crawl: Googlebot sends them back
GET /products/48213 HTTP/1.1
If-None-Match: "p-48213-v7"
If-Modified-Since: Tue, 02 Jun 2026 08:00:00 GMT
# Nothing changed: answer with no body at all
HTTP/1.1 304 Not Modified
ETag: "p-48213-v7"The empty body is the point. The server skips building the page, and nothing is transferred. Make the ETag change only when content a visitor would notice changes, not when a timestamp in the footer does.
Load fewer resources per page
Google's post on crawling page resources explains that the CSS, JavaScript and media fetched to render a page count against the crawl budget of the hostname serving them. Google's renderer caches those files for up to 30 days whatever your cache headers say, so stable resource URLs help and needless cache-busting parameters force refetches. Blocking render-critical files in robots.txt is still a mistake, because a page Google cannot render properly is a page it may not understand.
Raise demand where it counts
Link new and updated pages from pages Google already crawls often, such as category hubs and the home page, and keep XML sitemaps limited to canonical URLs with honest lastmod dates.
Changes that buy no crawling
Several common tweaks cost engineering time and change nothing. Googlebot ignores crawl-delay in robots.txt. Compressing sitemaps does not raise crawl budget. Touching pages so they look fresh adds no value, and changing URLs to force a recrawl wastes fetches when the content has not changed. Crawling is also not a ranking signal, so a site with more crawling than it needs gains nothing from more. The myths page linked above goes through each of these with Google's verdict.
Other bots and your crawl capacity
Third-party crawlers, from AI training bots to SEO tools and scrapers, do not spend Google's crawl budget. They do share your servers, though, and if their load slows responses or triggers errors, Google's capacity limit for your site goes down with it. So when an infrastructure team wants relief from bot traffic, the targeted answer is rate limiting or blocking the heavy bots by verified identity, not throttling every crawler at once.
Before blaming Googlebot for load, confirm the requests come from Google, using reverse DNS or Google's published IP ranges, since user agent strings are trivial to fake. If Googlebot itself has to slow down in an emergency, Google's guide to reducing crawl rate says to return 500, 503 or 429 for a short window of hours or a day or two. Keep it short. Serve those codes on the same URLs for several days and Google may drop them from the index.
Measuring it
Search Console's Crawl Stats report shows total crawl requests, download size and average response time, host status for robots.txt fetching, DNS and server connectivity, and breakdowns by response code, file type, crawl purpose (discovery or refresh) and Googlebot type. Read the trends rather than single days. Falling requests with rising response time point at capacity. A site that publishes every day but shows almost all crawling as refresh may have demand stuck on old URLs.
Server logs answer what Crawl Stats cannot: which URLs got the fetches. Filter to verified Googlebot, group URLs by pattern (products, default listings, sorted or filtered listings, internal search, redirects, errors) and answer four questions. What share of fetches lands on patterns you want indexed? How long do new URLs wait for a first crawl? How often are the pages that matter refreshed? How much goes to redirects and errors? This primer on log file analysis covers getting access to logs and verifying Googlebot.
Start by checking the three cases against your own site. If none applies, leave crawl budget alone and work on the pages. If one does, group a week of verified Googlebot requests by URL pattern before changing anything, because that table tells you which lever to pull first.