A practical guide to log file analysis for SEO

Log file analysis means reading your server's access logs to see which URLs search engine and AI crawlers requested, what status code each request got, and how often they came back. Get 30 to 90 days of logs with the client IP, timestamp, full URL, status code and user agent, keep only the requests whose IP matches the operator's published ranges, and group those verified hits by site section: that one table shows where crawling goes, where it fails, and which pages Google barely visits.

A desktop crawler such as Screaming Frog models what a bot could reach by following links. Logs record what bots requested, including URLs nothing links to anymore, and they are the only record of how AI crawlers treat your pages.

Reading one log line

Most web servers write some variant of the combined log format. Here is a made-up nginx entry for a Googlebot Smartphone request, with two custom fields appended at the end:

66.249.66.1 - - [09/Dec/2025:06:25:14 +0000] "GET /shoes/trail?color=blue&sort=price HTTP/1.1" 200 51234 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 0.184 HIT

The default combined format has no response time, no hostname and no cache status. On nginx you add them through the log_format directive, for example with the request time variable. Add the host as well if one server handles several hostnames, or www, non-www and subdomain traffic will blur into one set of paths.

Picking the layer you trust

Logs exist at several layers, and the layer you pick changes the numbers.

A CDN answers cached requests at the edge, so the origin server never sees them. Behind a CDN with a high cache hit rate, origin logs can miss a large share of bot requests, which makes edge logs (Cloudflare Logpush, Fastly real-time log streaming, Akamai DataStream and similar) the better source for crawl analysis. Load balancer logs sit in between. Managed hosting platforms vary: some expose raw logs or an export, some offer only aggregated analytics, and some offer nothing, which is good to know before anyone promises an analysis.

Choose one layer as the source of truth. If you stack edge and origin logs into one table, every cache miss appears twice, once per layer, and uncached templates look as if they get double the crawling. When you need both (edge for volume, origin for response times), join them on a request ID instead of appending one to the other.

Ask two more questions before you start. Does the pipeline sample? Some setups keep only a fraction of requests to save storage, which breaks every count and hides URLs that get only occasional visits. And how far back does it go? Thirty days is the minimum; 90 days shows sections Google visits only every few weeks.

Personal data and retention

Access logs hold IP addresses, which can be personal data under laws such as the GDPR, and query strings sometimes carry email addresses or session tokens. Agree a retention period with whoever owns privacy at your organization. Once the bots are verified, drop or truncate the IPs of everything else; an SEO analysis needs human traffic only as a total, if at all.

Verifying that Googlebot is Googlebot

User agent strings cost nothing to fake, so count a request as Googlebot only after checking its IP. Google sorts its crawlers and fetchers into three groups, each with its own reverse DNS pattern and its own published IP list, as its verification documentation describes:

For a handful of IPs, DNS works: run a reverse lookup on the IP, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP. For millions of rows, match against the JSON range files linked from the same page. Download every file the page lists, not only the one for Googlebot, so a user-triggered fetch gets labeled as Google rather than as an impostor, and refresh the files on a schedule because the ranges change.

The files from Google, OpenAI and Perplexity share one shape, a list of IPv4 and IPv6 prefixes, so one short script can label every request. A minimal Python version for a combined-format log:

import ipaddress, json, re, sys
from collections import Counter
from functools import lru_cache

# One JSON file per crawler group, saved from each operator's docs.
RANGES = {
    "google-common": "ranges/google-common.json",
    "google-special": "ranges/google-special.json",
    "openai-gptbot": "ranges/gptbot.json",
}

def load(path):
    with open(path) as f:
        prefixes = json.load(f)["prefixes"]
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
            for p in prefixes]

NETS = {name: load(path) for name, path in RANGES.items()}

@lru_cache(maxsize=None)
def owner(ip):
    addr = ipaddress.ip_address(ip)
    for name, nets in NETS.items():
        if any(addr in net for net in nets):
            return name
    return "unverified"

# Combined log format: ip - user [time] "METHOD url PROTO" status bytes "ref" "ua"
LINE = re.compile(r'(\S+) \S+ \S+ \[[^\]]+\] "\S+ (\S+) [^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')

hits = Counter()
for line in open(sys.argv[1]):
    m = LINE.match(line)
    if not m:
        continue
    ip, url, status, ua = m.groups()
    if "Googlebot" in ua:
        who = owner(ip)
        label = who if who.startswith("google") else "fake-googlebot"
        hits[(label, status)] += 1

for (label, status), n in hits.most_common():
    print(label, status, n)

Extend the same check to every bot you report on. Requests that claim to be a known crawler but come from outside its ranges go in a separate bucket; mixed in, they distort every other number.

Signed requests

IP lists are a workaround for the lack of real authentication. The IETF chartered a Web Bot Auth working group in October 2025 to standardize a way for bots to sign their requests cryptographically, building on HTTP Message Signatures (RFC 9421), so a site can check a bot's identity without maintaining IP lists. Adoption is early. Treat a valid signature as extra evidence and keep IP verification as the baseline.

AI crawlers need their own buckets

AI operators run several agents with different jobs, and the job decides what the traffic means. A training crawler collecting pages in bulk, a search crawler building an index for an answer engine, and a fetcher loading one page because a person asked about it should never share a line on a chart. This is what the operators themselves document:

OpenAI's crawler documentation explains the ChatGPT-User exception plainly: because those visits are initiated by a user, robots.txt rules may not apply. Perplexity says the same of Perplexity-User. Google's user-triggered fetchers work on the same logic, which is why they ignore robots.txt.

Google-Extended is missing from the table on purpose. Google's common crawlers page states that it has no separate HTTP user agent string: Google crawls with its existing user agents, and the Google-Extended token in robots.txt only controls whether that content may be used for Gemini model training and grounding. You will never find it in a log, and blocking it does not affect inclusion in Google Search.

For each AI agent, track volume over time, the sections it requests, the share of requests that pass IP verification, and whether it fetches paths your robots.txt disallows for it (keep a robots.txt history, since a new rule applies only after the bot refetches the file). Allowing or blocking these agents is a business decision. The robots.txt rules that implement it are the easy part; the logs tell you whether the rules hold.

Questions the data can answer

Start by bucketing verified Googlebot hits by template. A query like this, run in DuckDB against a CSV of verified hits, produces the base table the rest of the analysis builds on:

SELECT
  CASE
    WHEN url LIKE '/p/%'                        THEN 'product'
    WHEN url LIKE '/shoes/%' AND url LIKE '%?%' THEN 'category, filtered'
    WHEN url LIKE '/shoes/%'                    THEN 'category'
    WHEN url LIKE '/blog/%'                     THEN 'blog'
    ELSE 'other'
  END                  AS template,
  status,
  count(*)             AS hits,
  count(DISTINCT url)  AS distinct_urls
FROM read_csv_auto('verified_hits.csv')
WHERE bot = 'google-common'
GROUP BY ALL
ORDER BY hits DESC;

Is crawling going where the value is?

Put each template's share of verified Googlebot hits next to its share of organic landing pages or revenue. Say a shop finds that filtered category URLs absorb most of Googlebot's requests while product pages, which earn most of its organic revenue, get a small slice. That mismatch is the first project. Internal links, parameter handling and robots.txt rules decide where the crawl goes next, while canonical tags consolidate signals between duplicates but do not stop Googlebot from fetching them.

Which status codes are bots getting?

Trend status codes by template, week by week. Persistent 404s mean something still links to or lists the dead URLs, often an internal link or a stale XML sitemap. Large volumes of 301s point to internal links, sitemaps or old backlinks that still use outdated addresses. Clusters of 5xx or 429 responses need attention fast, because Google's crawlers slow down when they receive them. Also look for 304 responses: if your server supports conditional requests through ETag or Last-Modified headers, Googlebot can receive 304 Not Modified for unchanged pages instead of downloading them again. No 304s at all often means the server ignores those headers.

How deep into parameter space does Googlebot go?

Group requested URLs by their parameter names (sorted, values stripped) per template. Facets, sort orders, session IDs and calendar widgets can create near-endless URL sets, and crawl tools are often set to ignore parameters, so the logs may be the only place this waste shows up.

Which URLs does Google request that the site no longer links to?

Diff the URLs in verified Googlebot hits against a fresh crawl of the site. URLs that appear in the logs but not in the crawl are leftovers: old campaigns, removed sections, addresses from external links or old sitemaps. Decide per pattern. Relink the ones that should rank, redirect the ones with a clear equivalent, and let the rest return 404 or 410. The reverse set matters too: pages your site links to that Googlebot never requested.

How long between visits?

For each template, compute the median number of days between verified Googlebot requests to the same URL. If key product pages wait weeks between visits while tag archives are fetched daily, check internal linking and sitemaps first, since they are the main signals you control about which URLs matter.

Which Google crawler is it?

Split Google traffic by user agent instead of treating it as one bot. Googlebot Smartphone should account for most HTML crawling on a site indexed mobile-first, so a large desktop Googlebot share deserves a closer look. AdsBot, Google-InspectionTool (behind URL Inspection and the Rich Results Test) and the user-triggered fetchers say nothing about organic crawl capacity, so keep them out of crawl budget figures. Check requests for CSS and JavaScript files too: bundles that return errors to Googlebot break rendering, one of the failures covered in this guide to JavaScript SEO.

Watching a migration through the logs

During a migration or a large redirect project, logs show Googlebot's reaction within hours. Before launch, save a baseline: hits per template, the status code mix, and the URLs Googlebot requests most. After launch, check daily for the first weeks.

• Old URLs should return a single 301 to the right target, and Googlebot's requests for them should taper off over the following weeks.

• New URLs should start appearing in verified Googlebot hits soon after launch; if they stay absent, check that sitemaps and internal links point to them.

• A rise in 404s or 5xx responses on new templates is a release bug until proven otherwise.

• A 301 pointing to a URL that itself returns 301 is a chain; fix it at the source.

• robots.txt should return 200 on every request, because Google slows or stops crawling when it cannot get a usable robots.txt response.

Search Console's Crawl Stats report next to your logs

The Crawl Stats report shows the same activity from Google's side: requests by response code, file type, purpose and Googlebot type, plus host status for robots.txt fetches, DNS and server connectivity. Three differences from logs matter. It exists only for root-level properties, meaning a Domain property or a URL-prefix property at the root. Google says it reports most crawl requests but some might not be counted, so totals can differ slightly from your own. And its URL lists are representative examples, not complete inventories.

Use it to sanity check your pipeline: if your verified Googlebot totals and Google's are far apart, suspect the verification step or a missing log source. Use the logs for anything that needs every URL, and for every crawler that is not Google.

A monthly routine

Save the queries from the first analysis and rerun them each month. The routine fits in a short checklist:

1. Refresh the IP range files for every operator you track and re-run verification.

2. Compare verified Googlebot hits per template with the previous month.

3. Check 5xx and 429 counts per template and investigate any day with a spike.

4. List parameter patterns that appeared for the first time.

5. Review the URLs most often returning 404 or 301 to bots and trace where they are linked or listed.

6. Diff log URLs against the latest crawl to catch new orphans.

7. Count requests that claim a crawler user agent but fail verification, grouped by the claimed bot.

8. Note any new AI user agents and read their operator's documentation before writing robots.txt rules for them.

9. Record releases, migrations and robots.txt changes alongside the numbers, so next month's changes have an explanation.