Automating SEO monitoring with AI agents

Hand an agent the checks that have a fixed data source, a predictable update rhythm and a threshold you can write down as a number. Core Web Vitals field data, indexing progress for the URLs you care about, and the progress of a move to new infrastructure all qualify, while working out why something moved, and deciding what to do about it, stays with a person.

I started building monitoring agents for two reasons: to save time, and to have accurate answers delivered to my Slack. Mine check real-world CrUX data, the progress of indexed pages, and the progress of a switchover to new infrastructure. They compare what they find against yesterday's data and against weekly patterns. Anything below in the first person describes that setup. The rest is general guidance on data sources, quotas and alert design that applies to any agent.

Signals, sources and triggers

The table maps each signal to its data source, how often that data changes, and the shape of a rule that should get someone's attention. The thresholds in the last column are placeholders that show the form of a rule. Yours should come from your site's normal range.

CrUX field data and its 28-day window

CrUX supplies the field data in PageSpeed Insights and in Search Console's Core Web Vitals report, so it's the version of real-user experience that Google reports back to you. Core Web Vitals are judged at the 75th percentile of page loads: good means LCP within 2.5 seconds, INP of 200 milliseconds or less, and CLS of 0.1 or less.

The CrUX API documentation sets the constraints an agent has to live with. Data refreshes daily around 04:00 UTC on a best-effort basis, runs about two days behind today, and every record is a 28-day rolling aggregate. You can query an origin or a single URL and split the result by phone, tablet or desktop. The quota is 150 queries per minute per Google Cloud project, free of charge, and the docs state that it can't be raised by paying.

That 28-day window shapes the alert rule more than anything else. Each refresh adds one day and drops one, so a release that slows LCP today nudges the p75 a little each day for about four weeks instead of jumping overnight. A rule that waits for one big day-over-day jump can sit silent while a regression builds. A rule that fires on a threshold crossing, or on the same direction several days running, catches the climb. If you need to know within hours whether a deploy hurt, that job belongs to your own real-user monitoring or to lab tests, and the difference between those sources is covered in field data, lab data, and why your speed scores disagree.

The same window means weekday effects mostly cancel out, because 28 days is exactly four weeks. Comparing a CrUX reading with the same weekday last week adds little. Comparing it with yesterday, and with the trend, does.

Two quirks need to be spelled out for the agent. Pages below CrUX's popularity threshold have no record at all, and Google doesn't publish that threshold, so an empty response should be reported as "no data" and never treated as a pass. And when a URL that had a record suddenly loses it, someone should look: CrUX's eligibility rules also drop pages that return a non-200 status or carry noindex, so a missing record can be the first sign of a broken page rather than a traffic dip.

For the longer view, the CrUX History API returns weekly collection periods, each covering 28 days, refreshed each Monday around 04:00 UTC. It returns 25 periods by default and up to 40 on request, and shares the same per-project quota. That's enough for a weekly agent to answer "is mobile INP better or worse than a quarter ago?" without storing anything yourself. When INP is the metric that moves, this explainer on INP covers where interaction time tends to go.

Indexing progress without a Page indexing API

The Search Console API has no endpoint for the Page indexing report, so an agent that tracks indexing works one URL at a time through URL Inspection, within two limits. First, the method reports only the version in Google's index: the URL Inspection API reference says you can't test the indexability of a live URL with it. Second, the Search Console API usage limits cap inspections at 2,000 per day and 600 per minute for each site.

Each inspection returns a verdict, a coverage state, the last crawl time, the canonical Google selected, the canonical you declared, and whether robots.txt or a noindex rule blocked the page. Those fields turn into alerts an engineer can act on. A priority URL whose Google-selected canonical differs from the declared one, or whose last crawl time hasn't moved in weeks, points to something specific to open.

Here is a hypothetical worked example. Say a site launches a new section with 1,500 URLs. That fits inside one day's quota, so the agent can inspect every URL daily and post one line: how many are indexed today, yesterday and on the same weekday last week, plus the most common coverage states among the rest.

Now say the section has 50,000 URLs. Draw a random sample of 1,500 once, and inspect the same URLs every day. A fixed random sample of that size estimates the indexed share of the whole section to within about 2.5 percentage points either way, at 95% confidence. Better still, any day-to-day change inside the sample is a change in those actual URLs, not sampling noise. Redrawing the sample every morning throws that advantage away.

Leave the site-wide view to a person. Google's help page for the Page indexing report tells site owners not to expect every URL on a site to be indexed, and someone reading the reasons once a week will spot a new pattern faster than a count will. When the crawled but not indexed group starts growing, this guide to crawled but not indexed pages walks through the usual causes.

Infrastructure switchovers

My agents also track the progress of a switchover to new infrastructure. Moving hosting, a CDN or the serving stack behind the same URLs is close to what Google calls a site move without URL changes, and its guide to that kind of move lists what to watch: server logs on both the old and new servers, traffic falling on the old side as it rises on the new, the URL Inspection tool to confirm Googlebot can reach the new infrastructure, and firewalls that might block Googlebot. It also warns that a temporary drop in Googlebot's crawl rate right after launch is normal, and that the rate may later climb above where it started.

During that window, an agent can usefully measure the following.

• The share of requests served by the new stack, counted separately for users and for Googlebot verified by reverse DNS. A response header that names the serving stack makes this easy to count from logs and to spot-check with a plain HTTP request.

• The status code mix and response time on each stack, so a rise in 5xx errors or timeouts on the new side shows up before it reaches Search Console.

• Parity on a fixed list of URLs while both stacks are live: the same status code, title, canonical, robots meta tag and main content on old and new.

• Crawl rate, with a grace period written into the rule so the expected post-launch dip doesn't trigger an alert every morning.

Most of that work runs on access logs, and this primer on log file analysis covers parsing and bot verification. The same share-by-stack view works for AI crawlers if you split them out, and the guide to AI crawlers lists the user agents to look for. When URLs change as well, the job grows into a full migration with redirect mapping, which the site migration checklist covers step by step.

Yesterday and the same weekday

The two baselines my agents use, yesterday's data and weekly patterns, do different jobs. The yesterday comparison catches breaks, such as a template that lost its canonical overnight or a stack that stopped taking traffic. The weekly comparison handles rhythm. On many sites, crawling, publishing and visits follow a weekday pattern, so a Monday set against a Sunday looks like a jump that isn't one. Comparing with the same weekday last week removes most of that effect, and the median of the last four same weekdays is steadier still, because one public holiday can't drag it.

Which baseline fits depends on the signal.

Alerts people keep trusting

Keeping false alarms down means I have to keep monitoring what the agents send and keep iterating on it. I also have to think through exactly what I want from each alert, because otherwise the AI creates generic alerts.

Google's book on site reliability engineering states the cost of getting this wrong.

When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts.

That line comes from the monitoring chapter of Google's Site Reliability Engineering book, which also argues that every page should be actionable. An SEO alert is actionable when it names the metric and its scope, gives the current value and the baseline, says how fresh the data is, and points to the first thing to check. Here is a template, with every value hypothetical.

CWV ALERT (hypothetical values)
Mobile p75 INP on tracked product URLs crossed 200 ms

Metric: p75 INP, PHONE, CrUX API, URL level
Scope: 18 of 25 tracked product URLs; origin steady at 170 ms
Last 4 daily readings (median of the 18): 186, 192, 199, 204 ms
Trend: 172 ms in the History API period four weeks earlier
Rule: median above 200 ms, or worse 3 days in a row
Data window: 28 days ending two days before today
Check next: product template releases in the last 10 days,
then a lab trace of the add-to-cart click

The message carries its own evidence, so a reader can judge it without opening a dashboard. Decide as well what a quiet day looks like: either no message at all, or one line confirming the checks ran. The second option tells you the agent is still alive.

Prompting an agent for specific alerts

Ask a language model to "monitor SEO and alert on issues" and it will write alerts that read like a generic audit. Specific alerts come from specific instructions.

1. Name the source and the exact field, such as p75 INP for phones from the CrUX API, never "page speed".

2. Give the threshold and the baseline as numbers and dates.

3. Fix the scope: the URL list, the form factor, and whether to read origin or URL level data.

4. Require the evidence in the message: the current value, the baseline value and the data window.

5. List the known noise: CrUX's two-day lag and 28-day smoothing, the crawl-rate dip after a switchover, weekday rhythm and planned releases.

6. Say what to do with missing data: report "no record" and stop, never estimate.

7. Ban suggestions the data can't support. An indexing alert should not end with tips about page speed.

8. Define the quiet-day message.

9. Log every alert along with whether anyone acted on it, and adjust thresholds from that log.

Jobs that stay with a person

An agent that reads APIs and logs can tell you that something moved. The jobs below need context it doesn't have, or carry consequences that call for a human decision. This division is general guidance.

• Deciding whether a change was intended. A section noindexed on purpose produces the same signal as a bug.

• Diagnosis that needs release notes, code, or a conversation with the team that shipped the change.

• Rollbacks, redirects, robots.txt edits and anything else that writes to production.

• Explaining to stakeholders what happened and what it means for them.

• Setting thresholds, and retiring alerts nobody acts on. The SRE chapter makes the same point about signals that feed no dashboard and no alert: they are candidates for removal.

To start, pick one of the three signals, write its rule as a single sentence with a number and a baseline, and let it run for two weeks while you mark each alert as useful or noise. Add the next signal once the first one has earned its place in the channel.