A practical primer on log file analysis for SEO
Crawl tools tell you what a bot could do on your site. Log files tell you what Googlebot actually did. The gap between those two is where most interesting technical SEO problems live.
Log analysis may be the single highest-leverage skill separating decent technical SEOs from great ones, and it is far less intimidating than it sounds. This primer covers everything needed to get started.
Why logs beat crawl tools
Screaming Frog simulates a crawler starting from your homepage and following links. Useful, but it is a model. Logs are ground truth: every request Googlebot made, when it made it, what status code it got back, and how the server behaved under real conditions.
Only logs can answer questions like: which sections does Googlebot actually visit daily versus quarterly? Is it burning half its requests on faceted parameter URLs? Is it still hammering URLs you redirected two years ago? Does it fetch pages that no crawl tool ever finds because nothing links to them anymore? No simulation gets you this.
Getting your hands on logs
This is genuinely the hardest step, and it is political as much as technical. Where the logs live depends on the stack:
CDNs: Cloudflare Logpush, Fastly log streaming, Akamai DataStream. Critical detail: edge logs capture requests the origin never sees because the CDN answered from cache. If you only have origin logs behind a CDN, you are missing most bot traffic.
Reverse proxies and web servers: nginx and Apache access logs, the classic format. Confirm the user agent and, ideally, response time fields are enabled.
Load balancers and cloud front doors: ALB logs to S3, Cloud Load Balancing to BigQuery, and so on.
Managed platforms: many expose logs through an API or a built-in analytics export; some simply do not, which is worth knowing before you promise the analysis.
The fields you need: timestamp, full URL including query string, status code, user agent, client IP, and response size or time if available. Ask for 30 days minimum; 90 is better for spotting patterns on sections Google visits rarely. And ask about sampling. Some log pipelines keep one request in ten, which quietly wrecks any counting you do downstream.
Verify it is really Googlebot
A depressing share of traffic claiming to be Googlebot is scrapers wearing its user agent. Before you analyze anything, verify. Google publishes its crawler IP ranges as JSON files (one for Googlebot proper, another for special crawlers), so the scalable method is a join between your log IPs and those published ranges. The classic alternative is reverse DNS lookup confirming a googlebot.com or google.com hostname, then a forward lookup confirming the IP matches.
Do the same for Bingbot and for the AI crawlers; operators such as OpenAI and Perplexity publish their IP ranges, and others document verification methods, so check each bot's official documentation. In most real datasets, dropping unverified impostors changes the numbers enough to change the conclusions.
What to actually look for
Crawl distribution by section
Bucket verified Googlebot hits by URL pattern: product pages, categories, blog, parameters, assets. Then put that next to where your value lives. When 40% of crawl activity lands on a template that drives 2% of your organic revenue, you have found your first project.
Status codes served to bots
Group by status code and trend it over time. Persistent 404 crawling tells you dead URLs still have links or sitemap entries pointing at them. Heavy 301 traffic means internal links or old references still route through redirects. Any 5xx cluster is urgent: if Googlebot repeatedly hits server errors, it slows down crawling to protect your site, precisely what you do not want. Also check whether you serve 304s to conditional requests; on large sites, telling Google 'nothing changed' is a genuine efficiency win.
Parameter traps
Sort crawled URLs by query string patterns. Faceted navigation, sort orders, session IDs, and calendar widgets generate infinite URL spaces, and logs show you exactly how deep Googlebot has wandered in. Audits regularly turn up logs where a filter combination explosion was absorbing the majority of all bot requests, invisible to every crawl tool because the crawl was configured to ignore parameters.
Orphan URLs
Diff the set of URLs Googlebot requests against the set your own crawl discovers. URLs in the logs but absent from the crawl are orphans: pages Google remembers from old links, old sitemaps, or external references, that your current site no longer links to. Some deserve reintegration into the internal linking; others deserve a clean 410.
Crawl frequency vs importance
For each key template, compute average days between Googlebot visits per URL. Money pages crawled monthly while tag archives get crawled daily is a misallocation signal, and usually reflects your internal linking more than anything else. Google crawls what your architecture emphasizes.
Googlebot variants
Do not lump every Google user agent into one bucket. Googlebot Smartphone should dominate now that mobile-first indexing is long complete; if the desktop crawler still leads on your site, find out why. Googlebot-Image and Googlebot-Video volumes tell you how seriously Google takes your media, and heavy AdsBot or Google-InspectionTool traffic has entirely different explanations than organic crawling. Segmenting by variant regularly turns one confusing trend line into three obvious ones.
AI bot traffic is its own workstream
Logs from any reasonably visible site now show a zoo of AI agents: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended, Bytespider, and a long tail of newcomers. Treat them as a separate analysis with separate questions.
Distinguish training crawlers from user-triggered fetchers. A training bot bulk-collecting your content and an agent fetching one page because a user asked a question about your product are different phenomena with different policy implications. Measure each bot's volume, which sections it favors, and whether it respects your robots.txt. Then verify against published IP ranges where available, because AI user agents are now the favorite costume for scrapers. What you do with the answers is a business decision, but you cannot make it without the measurements, and the logs are the only place they exist.
A practical workflow
Pull 30 to 90 days of logs into one place. For moderate volumes, DuckDB or BigQuery handles millions of rows without complaint; Screaming Frog's Log File Analyser works well for smaller sets.
Parse and normalize: timestamp, URL, status, user agent, IP. Lowercase hostnames, keep query strings intact.
Verify bots against published IP ranges and tag each request: Googlebot, Googlebot-Image, Bingbot, each AI bot, unverified impostor.
Segment URLs by template using path patterns, the same buckets you would use in an analytics report.
Build the core reports: hits by section over time, status codes by section, top parameter patterns, orphan candidates, and per-template crawl recency.
Run a fresh Screaming Frog crawl and join it to the log data to separate 'Google crawls it but we do not link it' from 'we link it but Google skips it'.
Turn findings into tickets with the log evidence attached. 'Googlebot spent 38% of its requests this month on filter combinations' moves an engineering roadmap in a way 'we should improve crawl efficiency' never will.
Then re-run it after the fixes ship. Logs are the only channel where you can watch Googlebot change its behavior in response to your work, sometimes within days. Search Console's Crawl stats report gives you a sampled preview of this, and it is a fine appetizer, but the logs are the meal.
Start smaller than you think
You do not need a data pipeline to begin. One month of logs, one afternoon, and a handful of group-by queries will surface something actionable on almost any site of real size: a redirect loop Googlebot visits hourly, a parameter swamp, a section it barely knows exists. In most audits, the logs contradict at least one thing everyone believed about how Google crawls the site. Finding out which thing is the fun part.
And once the first analysis pays off, make it repeatable. A saved set of queries re-run monthly costs almost nothing and turns log analysis from a heroic one-off into an early warning system: 5xx creep, a new parameter explosion, an AI bot that tripled its appetite. The sites that catch crawl problems early are never the ones with the fanciest tooling. They are the ones that keep looking.