XML sitemaps that help discovery and diagnosis

An XML sitemap is a hint: it helps Google find URLs and decide when to recrawl them, but it cannot force indexing or lift rankings. The sitemaps that earn their keep list only canonical, indexable URLs that return 200, carry a lastmod date that moves only when the content does, and are split by page type so Search Console can show which template Google is passing over.

Not every site needs one. Search Console's Sitemaps report documentation says a site of about 500 pages or fewer, where every page can be reached by following links from the homepage, probably does not. Large sites, new sites with few external links, sites with sections that are poorly linked, and sites that publish often are the ones that benefit.

The parts of a sitemap Google uses

Google's guide to building a sitemap is short and specific about which parts of the format it uses:

The same guide calls submission "merely a hint": Google may not download the file, and listing a URL does not mean it will be crawled. A URL stuck in Discovered, currently not indexed was found, which is the part a sitemap controls. Whether it gets crawled and indexed depends on the page and the site.

A lastmod policy engineers can implement

lastmod is the one optional field with a payoff, and it is easy to ruin. When Google retired the sitemap ping endpoint in 2023, it also explained how it treats lastmod: as a signal for scheduling recrawls of known URLs, provided the dates match reality. A site that claims yesterday's date on pages that have not changed in years eventually stops being believed. The same post defines the change that counts as a "last significant modification" and says it is fine to leave lastmod out for pages whose real modification date the software cannot tell.

That turns into a policy table a development team can build against:

The implementation detail that makes this work is storing a "last significant change" timestamp in the CMS, separate from the database's updated-at field, and generating the sitemap from that column. Dates must use W3C Datetime format; a date alone is fine, but if you include a time, include the time zone too, or Search Console reports an invalid date.

Updating lastmod without changing the page does not help either. Google has said it rates content by quality regardless of age, and that making pages look fresh with trivial edits adds nothing.

Rules that sitemap generators get wrong

• Every URL must be absolute and use the same protocol and host as the sitemap file. An https sitemap listing http URLs, or a www sitemap listing non-www URLs, produces "URL not allowed" errors.

• Location sets scope. Unless you submit it through Search Console, a sitemap covers only URLs below its own directory, which is why Google recommends the site root.

• A single file holds at most 50,000 URLs or 50MB uncompressed. Past either limit, split the file and list the parts in a sitemap index.

• A sitemap index lists sitemaps only. Search Console rejects an index that points to another index (or to itself) as a nested sitemap index.

• Google's guide to sitemap index files adds that listed sitemaps must sit on the same site as the index, in the same directory or deeper, unless you have set up cross-site submission, and that Search Console accepts up to 500 index files per site.

• Files must be UTF-8 and entity-escaped, so an ampersand or quote mark inside a URL has to be written as its XML entity.

• Gzip saves your bandwidth, not Google's crawl effort. Google has said compressed sitemaps do not increase crawl budget, since the files still have to be fetched.

Split sitemaps by template

A single sitemap gives you one indexing number for the whole site. Separate sitemaps per page type give you one number per template, because Search Console's Page indexing report can be filtered to a specific sitemap or sitemap index, whether it was submitted in the Sitemaps report or referenced from robots.txt. That filter turns "some pages are not indexed" into "this template is not indexed".

Here is an index for a hypothetical retailer, kept in a /sitemaps/ directory so every child file sits in the same directory as the index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemaps/products-1.xml.gz</loc>
    <lastmod>2025-12-14T08:30:00+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/categories.xml.gz</loc>
    <lastmod>2025-12-12T17:05:00+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/guides.xml.gz</loc>
    <lastmod>2025-12-01T09:00:00+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/stores.xml.gz</loc>
    <lastmod>2025-11-20T10:00:00+00:00</lastmod>
  </sitemap>
</sitemapindex>

And the one line robots.txt needs, which any crawler that reads robots.txt can find:

Sitemap: https://www.example.com/sitemaps/index.xml

Submit the index in the Sitemaps report as well, because that report lists only sitemaps submitted there or through the API, never ones found in robots.txt, and it is where fetch failures and parse errors show up. If a child sitemap does not appear as its own option in the Page indexing filter, submit it individually.

Now say that retailer filters the Page indexing report sitemap by sitemap. Products, categories and guides are mostly indexed, but most store pages sit in Crawled, currently not indexed. The problem now has an address: the store template, where every page is a map embed, an address and the same paragraph. Without the split, the same pages would have been a vague dip in a site-wide total.

When one template outgrows 50,000 URLs, split it by something stable, such as ID ranges or creation year, so a URL stays in the same file for life. Avoid files built around "recently changed" URLs that churn every day. Google's crawling guidance specifically warns against rotating sitemaps to steer crawl budget.

Entries to keep out of the file

A sitemap entry is also a canonical hint: Google says it generally shows canonical URLs in results and that sitemaps can influence which URL it picks. Entries that contradict your other signals weaken that hint, which is why canonical tags and sitemaps need to agree. Remove:

• URLs that redirect, including http to https and trailing-slash redirects

• URLs returning 404 or 410, and pages Google reports as soft 404s

• Pages with a noindex tag or header

• Non-canonical variants such as sort, filter, tracking and session parameters

• URLs disallowed in robots.txt, since listing a page you block sends two opposite messages (the robots.txt guide covers why blocked URLs can still get indexed)

• Staging, preview or test hostnames that leaked in through shared configuration

• Pages that require a login

The Page indexing documentation names submitted sitemaps that include blocked, noindexed or login-protected URLs as a common cause of error spikes in the report, so a dirty sitemap also makes your monitoring noisier.

Media, news and alternate-language sitemaps

The core format can be extended in the same file or in separate ones. Image entries help Google find images it might otherwise miss, such as those loaded by JavaScript. Video entries describe videos on a page. News sitemaps are for news publishers and should list only articles created in the last two days, with up to 1,000 news entries per file.

Alternate-language versions can be declared in the sitemap instead of in each page's head. Each URL gets its own entry, and each entry lists every language or regional version, including itself, as an xhtml:link element with a full absolute URL. If two versions do not point to each other, Google ignores the pair. For sites with JavaScript-heavy templates, putting hreflang in the sitemap also takes it out of the rendering path, one of the problems covered in this guide to JavaScript SEO.

Several sites, one sitemap location

Companies that run several domains can host all sitemaps in one place. The build guide describes two routes: verify every site in Search Console and submit the sitemaps there, or keep one sitemap per site in the central location and reference it from that site's own robots.txt. In the robots.txt route, each sitemap may list URLs from its own site only.

Pings are gone, and IndexNow is a separate system

Google's sitemap ping endpoint stopped working six months after the 2023 announcement; requests to it return a 404 and do nothing. If a CMS plugin or deploy script still calls it, remove the call. Discovery now runs through the robots.txt Sitemap line, submission in Search Console or its API, and Google's own periodic refetching of known sitemaps.

IndexNow is a push protocol run by other search engines. The participating engines listed on indexnow.org include Bing, Yandex, Naver and Seznam; Google is not among them. Use it if those engines matter to you, and keep the sitemap for Google.

Sitemaps report errors and their usual causes

A sitemap with errors is still partly used: URLs that parse cleanly are queued for crawling. A sitemap that cannot be fetched at all is retried for a few days, after which Google stops trying until you fix and resubmit it.

A quarterly sitemap QA checklist

1. Crawl every sitemap URL in list mode and remove anything other than 200, indexable, self-canonical URLs.

2. Diff the sitemap against a full site crawl. Indexable pages missing from the sitemap point to generator gaps; sitemap URLs the crawl never reached have no internal links, which server log file analysis can confirm from Googlebot's side.

3. Compare lastmod with the last significant change on a sample of URLs per template.

4. Flag files where every URL shares the same lastmod, or where dates sit in the future; both point to a build stamping dates.

5. Review the Sitemaps report for fetch failures and parse errors.

6. Filter the Page indexing report by each sitemap and compare the indexed share with last quarter.

7. Confirm robots.txt still references the index and does not block any sitemap file.

8. Search the files for staging or test hostnames.

9. Remove any leftover ping calls from plugins and deploy scripts.

The first two checks also belong in release QA for any change that touches URL generation, since that is when generators drift.