What a slow server costs your SEO, from crawling to LCP

A slow server costs you in two places. For visitors, every millisecond before the first byte arrives comes out of the 2.5-second budget for Largest Contentful Paint (LCP), a Core Web Vital; for Googlebot, slow or failing responses lower how much Google is willing to crawl, which matters for large or fast-changing sites. Most small sites only feel the first cost, but the order of work is the same for everyone: measure time to first byte (TTFB) for real users, check whether bots get a slower server than people do, then fix the slowest link in the chain.

Everything that happens before the first byte

web.dev's definition of TTFB counts the whole span from the start of navigation to the first byte of the response, not only the time the server spends thinking:

navigation starts
  |-- redirects (each hop repeats most of the steps below)
  |-- service worker startup (if one is installed)
  |-- DNS lookup
  |-- TCP connection and TLS handshake
  |-- request sent
  |-- server work: routing, cache lookup, database and API calls, rendering
  |-- first byte arrives   <- TTFB ends here (a 103 Early Hints response counts)

That breakdown decides where to look. A page can sit on a fast server and still have a poor TTFB because it is reached through two redirects, or because visitors connect from another continent with no CDN in between.

web.dev treats 0.8 seconds or less as good and more than 1.8 seconds as poor, and calls these a rough guide, since TTFB sits outside the Core Web Vitals. The same page makes a point that matters for SEO decisions. A server-rendered site can have a higher TTFB but better First Contentful Paint and LCP than a client-rendered one, because the HTML arrives ready to paint. Dropping server rendering to win back TTFB trades a diagnostic metric for the ones that count, and gives up the indexing benefits described in the JavaScript SEO guide.

The LCP arithmetic shows when the server is the bigger lever. Say a product template has a 75th-percentile TTFB of 1.6 seconds. That leaves 0.9 seconds to download the HTML, discover and fetch the hero image, and paint it, on the slower quarter of devices and connections. Front-end work can only shave pieces off that 0.9 seconds. A template with a 0.4-second TTFB and a 3.5-second LCP, on the other hand, has a front-end problem, and server work will not fix it.

Two systems that react to a slow server

Ranking, through Core Web Vitals

Google's page experience documentation says "Core Web Vitals are used by our ranking systems," and also that Search "always seeks to show the most relevant content, even if the page experience is sub-par." As far as Google documents it, TTFB reaches ranking only through LCP. A faster server will not lift a page that answers the query worse than its competitors, but a server slow enough to push LCP past 2.5 seconds is a problem you can measure and fix.

Crawling, through crawl capacity

Here the effect is direct. Google's crawl budget guide describes a crawl capacity limit that rises when a site responds consistently and its response times stay stable or improve. If the site slows down, or answers with 5xx server errors or rate-limiting signals such as HTTP 429, "the limit goes down and Google crawls less." A bot-protection rule that answers Googlebot with 429 during a traffic spike is read the same way, as a reason to crawl less.

The same guide is clear about who needs to care: sites with more than a million unique pages that change about weekly, sites with more than 10,000 pages that change daily, and sites with a large share of URLs that Search Console reports as discovered but not yet indexed. Google calls those numbers rough estimates. A 300-page brochure site is unlikely to see its crawl coverage suffer from a sluggish server, while a marketplace adding thousands of listings a day can see new pages wait longer to be crawled. The post on crawl budget myths and realities covers that line in more detail.

Search Console's Crawl Stats report shows the bot side, with two details that are easy to miss. Its average response time covers all resources fetched from your site, not only HTML pages, so a slow image server or API endpoint moves the line too. And the report is aimed at advanced users; Google says a site with fewer than a thousand pages should not need it. Its help page states the mechanism plainly: "If your site is responding slowly to requests, Googlebot will throttle back its requests to avoid overloading your server."

Bots can get a slower server than people

Real-user TTFB and bot TTFB can diverge widely on the same site, for structural reasons.

Googlebot crawls from far away for many sites. Google's documentation on locale-adaptive pages says "the default IP addresses of the Googlebot crawler appear to be based in the USA." An origin in Europe or Asia without a CDN adds long round trips to every crawl request, which local visitors never pay.

Bots also go where people do not. Caches fill according to human traffic, and crawlers spend much of their time on the long tail.

Say a shop has 40,000 product URLs whose stock and prices change daily, behind a CDN that caches HTML for ten minutes. Shoppers concentrate on a few hundred bestsellers, so almost every human page view is served from the edge and the CrUX TTFB for the site looks healthy. Googlebot's requests spread across all 40,000 URLs, and most arrive when no fresh cached copy exists, so the origin renders each one from scratch. The Crawl Stats report and the server logs then show response times several times higher than anything users experience, and the crawl rate on new listings slows accordingly. Nothing on the user-facing dashboards hints at the problem.

To see this on your own site, use logs that record response time and cache status. Filter to Googlebot requests verified by reverse DNS, then compare their response times with human requests for the same templates, split by cache hit and miss. The log file analysis primer covers getting the logs and verifying the bots.

Matching symptoms to causes

Measuring it yourself

Field data comes first. CrUX reports TTFB at the 75th percentile in PageSpeed Insights, and the web-vitals library can collect it in your own RUM; the post on field data, lab data and RUM explains why the two will not match exactly.

For spot checks, curl shows TTFB and the cache headers in one go. Run each URL a few times so the first request can warm the cache, and include a URL that bypasses the cache, which web.dev's TTFB optimization guide recommends so you can see the full server time a cache would otherwise hide. A query parameter works as a bypass only if your CDN includes the query string in its cache key.

for url in "https://example.com/" "https://example.com/products/blue-widget?nocache=1"; do
  for i in 1 2 3; do
    curl -s -o /dev/null -D - \
      -w "ttfb=%{time_starttransfer}s total=%{time_total}s\n" "$url" \
      | grep -iE "^(age|x-cache|cf-cache-status|server-timing):|^ttfb="
  done
done

The cache header names vary by CDN; age, x-cache and cf-cache-status are common ones. To see inside the server's time, add a Server-Timing header to responses. The same guide describes it: each entry names a step and its duration, and the values show up in the DevTools Network panel and in the Navigation Timing API, so your RUM can collect them too.

Server-Timing: cache;desc="MISS", db;dur=212, render;dur=148

Fixes, roughly in order of payoff

The biggest single change is caching HTML at the edge for pages that are the same for every visitor, so most human and bot requests never reach the origin. This header lets browsers revalidate on every visit while shared caches keep a copy for ten minutes, serve the stale copy for up to a day while fetching a fresh one in the background, and keep serving it if the origin starts failing:

Cache-Control: public, max-age=0, s-maxage=600, stale-while-revalidate=86400, stale-if-error=86400

Three cautions come with it. Never cache personalized or logged-in HTML in a shared cache. Purge or shorten the cache life for pages that change on publish, such as listings and prices. And check how your CDN handles s-maxage and the stale directives, because support differs between providers. The stale-if-error directive has a side benefit for crawling: when the origin fails, users and bots keep getting the last good copy with a 200 instead of a 5xx.

After edge caching, the remaining fixes in rough order:

1. Cache expensive data on the server, such as CMS queries and API responses, and run independent fetches in parallel instead of one after another.

2. Remove redirect hops by pointing internal links, canonical tags and sitemaps at final URLs. Chains built up over past moves are a common source, which is why the site migration checklist insists on one hop.

3. Bring the network closer to the audience with a CDN that terminates TLS at the edge and speaks HTTP/2 or HTTP/3.

4. Send something early. Streaming or flushing the head of the document gets bytes moving while the body renders, and a 103 Early Hints response lets the browser preconnect to origins and preload critical files during server think time.

5. Answer conditional requests with 304 Not Modified. The crawl budget guide asks for this, because it lets Google reuse its cached copy of an unchanged page instead of downloading it again.

6. Check the host itself: memory, shared-hosting contention, missing database indexes, and cold starts on serverless functions.

Knowing when to stop

If field TTFB at the 75th percentile sits comfortably under 0.8 seconds and bot response times in the logs look similar, the remaining LCP problems are in the front end: image size and priority, render-blocking resources, client-side rendering. Taking TTFB from 300 milliseconds to 250 will not move LCP enough to matter. A sensible first step is to run the curl loop above against a cached and an uncached URL from your three biggest templates, and compare the results with what your logs say Googlebot gets on the same templates.