Faceted navigation and pagination on large listing pages

Give an indexable URL only to the filter combinations people search for and that you can fill with products, keep every other combination out of the crawl with robots.txt or URL fragments, and paginate listings as plain linked pages that each keep their own canonical. That setup lets Google reach every product without spending its visits on filter URLs nobody wants.

Why filters multiply into millions of URLs

Take a hypothetical running shoes category with four single-select filters: brand (30 values), color (12), size (8) and price band (6).

That is 25,388 filtered listings for one category. Give each an average of two pages and five sort orders and the category alone produces about 250,000 crawlable URLs. Multi-select within a filter (red or blue), parameters that can appear in any order, and mixed-case values push the number far higher, and none of these URLs leads to a product the default listing does not already link.

Google's faceted navigation guide names the two costs. Crawlers cannot tell that a new filter URL is useless without fetching it, so they fetch a great many before giving up, and every fetch spent there is one not spent discovering new products. In Google's crawl budget guide, this kind of URL inventory is the factor a site can control most, and a large share of URLs in "Discovered - currently not indexed" is one of the signs the guide gives that a site has more URLs than Google is willing to crawl.

Google used to offer a URL Parameters tool in Search Console for telling it which parameters to ignore. It was retired in 2022, so containment now has to happen on the site itself.

Choosing which combinations get indexable URLs

A filtered listing earns its own indexable URL when it passes four checks:

1. People search for that combination as a phrase. "Nike running shoes" and "red running shoes" are searches; "running shoes, size 44, sorted by newest" is a filter state.

2. The catalog can fill it. Decide a minimum product count that makes the page a satisfying answer for your catalog, and enforce it in code, so a promoted page loses its links and sitemap entry when stock falls below the floor and returns 404 if it empties completely.

3. It exists most of the year. A combination that has products for two months will spend the other ten as a thin page, so seasonal pages should be a deliberate exception.

4. It answers a different question from its parent. Brand within category usually does. Size usually does not.

Applied to common filter types, those checks tend to produce defaults like these:

Keep the promoted combinations in a list someone owns, not in scattered template logic. Each one gets a clean path such as /running-shoes/nike/, a self-referencing canonical, its own title and heading, an entry in the XML sitemap, and links from the pages around it. Review the list on a schedule, demoting combinations whose demand or stock has faded and promoting new ones.

Keeping the rest out of the crawl

Each containment method controls something different, and Google documents the limits of each:

Google's guide recommends robots.txt or fragments when filter URLs do not need to be indexed. It describes canonical and nofollow as generally less effective in the long term, though it says a canonical pointing filtered URLs at the unfiltered version may reduce their crawling over time. A canonical is a hint, and Google can pick a different URL, for example when the filtered page's content differs a lot from its target. This post on how Google treats canonical tags covers when that happens.

Here is a robots.txt block for a site whose promoted combinations live on clean paths and whose other filters are parameters:

User-agent: *
# Sort, view and page-size variants never deserve a crawl
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?view=
Disallow: /*&view=
Disallow: /*?per_page=
Disallow: /*&per_page=
# Filters that are never promoted
Disallow: /*?price=
Disallow: /*&price=
Disallow: /*?in_stock=
Disallow: /*&in_stock=
# Color and size as parameters. Promoted combinations
# use paths such as /running-shoes/red/ and stay crawlable
Disallow: /*?color=
Disallow: /*&color=
Disallow: /*?size=
Disallow: /*&size=

Pairing each name with ? and & matches the parameter wherever it sits in the query string without catching longer names that end the same way. A looser rule such as "/*?*size=" also matches every parameter whose name merely ends in size, such as ?pagesize= or ?boxsize=, so it can block more than intended. The page parameter is left alone, so /running-shoes/?page=3 stays crawlable while /running-shoes/?page=3&sort=price does not. The robots.txt guide covers the matching rules in more depth.

Order matters if junk filter URLs are already in the index. Google's noindex documentation points out that a crawler blocked by robots.txt never sees a noindex, so blocking first can leave those URLs lingering as bare results. Let them be crawled with noindex, or answer them with 404 where they are empty, until they drop out, then add the Disallow.

Normalizing the URLs you keep

Every crawlable listing should have exactly one spelling. Use this as a checklist:

• Separate parameters with &. Google's guide says commas, semicolons and brackets are hard for crawlers to recognize as separators.

• Write filters in one fixed order, whether they sit in the path or the query string, and redirect any other order to it. Duplicate filters should not be possible.

• Lowercase values and give each value one spelling, so red, Red and RED are one URL.

• Drop empty parameters (?color=&size=) and defaults (?page=1, ?sort=relevance) from links and redirect them away.

• Return 404 at the URL itself when a combination has no results, contains a duplicate filter, makes no sense, or asks for a page past the end of the listing. Google's guide is explicit that an empty combination should not redirect to a generic not-found page.

• Keep tracking parameters out of internal links.

Links to promoted filter pages

A promoted filter page reachable only by clicking a filter widget looks marginal to a crawler. Link it from the parent category's copy, from a short block of popular filters, and from related categories where that makes sense to a shopper. Google's myths and facts about crawling says pages linked directly from the home page may be treated as more important and crawled more often, but that this does not make them rank higher, so treat link prominence as a crawl priority setting rather than a ranking trick. The same principles apply as for any internal linking at scale. What does not help is a footer block linking to every combination the template can generate.

Pagination on listing pages

Pagination is how crawlers reach products beyond the first screen, so it has to work without JavaScript and without guesswork. Google's pagination guide covers the essentials:

• Give every page in the sequence its own URL, with the number in a query parameter or the path, and link each page to the next with an ordinary a href link. Google also suggests linking every page back to the first one, as a hint about which page is the better landing page.

• Let each page use its own URL as the canonical. Pointing pages 2 and beyond at page 1 tells Google to ignore the only pages that link to deeper products.

• Reuse the same title and description across the sequence if you like. Google says that is fine, so adding "Page 3" is optional.

• Keep page numbers out of URL fragments. Google ignores everything after the #.

• Leave rel="prev" and rel="next" in place if the platform emits them. Google does not use them, but the guide notes other search engines may.

• Paginate the default listing and promoted filter pages only. Sorted and filtered variants should not spawn crawlable paginated series of their own.

• Back the listing up with an XML sitemap or a Merchant Center feed that lists every product, as Google suggests.

A crawlable pagination block for page 3 looks like this:

<!-- in the head of page 3 -->
<link rel="canonical" href="https://www.example.com/running-shoes/?page=3">

<!-- in the body -->
<nav aria-label="Pagination">
  <a href="/running-shoes/">First</a>
  <a href="/running-shoes/?page=2">Previous</a>
  <span aria-current="page">Page 3</span>
  <a href="/running-shoes/?page=4">Next</a>
</nav>

Page 1 links to /running-shoes/, not /running-shoes/?page=1, so the default listing keeps one URL.

Two older habits still come up. Adding noindex to deep pages does not save crawling, because Google must still fetch them to see the tag, and it adds nothing Google's guide asks for, so leave them indexable. A view-all page can work when the whole list is short enough to load fast on a phone, but Google's current pagination guide does not describe it, and a view-all listing of thousands of products is too heavy to serve anyone well. For large catalogs, self-canonical pagination is the safer default.

Infinite scroll and load more

Google Search does not scroll or click. Its lazy-loading guide says to give each chunk of an infinite list its own persistent URL, keep the content at that URL the same on every load, link the chunks sequentially, and update the address bar with the History API as a new chunk becomes the main thing on screen. In other words, infinite scroll should sit on top of the paginated URLs above, never replace them.

Two things make that work. First, /running-shoes/?page=4 must load directly as a server-rendered page with that chunk's products and links to pages 3 and 5. Second, the script that appends chunks keeps the URL in step:

// When a chunk is mostly in view, show its URL in the address bar
const observer = new IntersectionObserver((entries) => {
  for (const entry of entries) {
    if (!entry.isIntersecting) continue;
    const page = entry.target.dataset.page;
    const url = page === "1" ? "/running-shoes/" : "/running-shoes/?page=" + page;
    history.replaceState(null, "", url);
  }
}, { threshold: 0.5 });

document.querySelectorAll("[data-page]").forEach((chunk) => observer.observe(chunk));

replaceState keeps the back button from filling up with one entry per chunk. pushState works too if you want each chunk in the browser history.

A "Load more" control should be a real link to the next page URL that JavaScript intercepts, not a button with no href. Crawlers then see a link, and people benefit as well, because endless auto-loading keeps pushing the footer out of reach for keyboard users.

To test, fetch a deep page URL with JavaScript off, using curl or a crawler with rendering disabled, and confirm its products and its next link are in the raw HTML. Then check the rendered HTML of the same URL in URL Inspection. This guide to JavaScript SEO covers the wider testing routine.

A staged rollout

Blocking a parameter that Google was using as its only path to some products is the expensive mistake here, so roll changes out in stages:

1. Start with one category group or one parameter, not the whole site.

2. Before blocking anything, crawl the site with the planned rules applied and compare the product URLs found against a crawl without them. Any product that disappears needs another link path first.

3. If junk filter URLs are already indexed, clear them with noindex or 404 before adding the Disallow, as described above.

4. Change the templates so they stop generating links to contained patterns, and ship the normalization redirects and 404s with the robots.txt rules.

5. Watch verified Googlebot requests in your server logs. Requests to blocked patterns should fade, requests to products should hold or grow, and new products should get their first crawl sooner. This primer on log file analysis covers the setup.

6. In Search Console, watch "Indexed, though blocked by robots.txt" for the blocked patterns and the indexing of promoted pages.

7. Review the promoted list on a fixed schedule so the indexable set keeps matching what people search for and what is in stock.