Crawl Budget for E-commerce: Pagination, Filters and Faceted Navigation

Is Googlebot spending its time on ?color=red&size=m&sort=price_asc instead of your new product pages? Then your filters have become a crawl trap. Faceted navigation SEO is the work of deciding which filtered URLs should be crawled and indexed, and blocking the rest from crawling. It also means making sure pagination and internal links point Google at the pages that actually sell.

Here is the short version of what I recommend. Keep a small set of valuable filter pages indexable. Block low-value filter combinations in robots.txt. Give every paginated page its own canonical. Then check the results in your server logs and Search Console’s Crawl Stats report.

Why Filters Turn Into Crawl Traps

Filters are great for shoppers. The problem is that most platforms create a new URL for every filter a visitor clicks. And every one of those URLs is linked from the page Googlebot just crawled.

Do the maths for one category with 8 colours, 6 sizes, 5 brands, 4 price bands and 3 sort orders. If each filter can be on or off, that is 9 × 7 × 6 × 5 × 4 = 7,560 possible URLs for one category. Add multi-select filters, different parameter orders and page numbers, and there is no practical limit.

Google’s own documentation describes two effects:

• Overcrawling. A crawler cannot know a filtered URL is useless until it fetches it. So it may request a huge number of them before it learns the pattern.
• Slower discovery. Time spent on near-duplicate filter pages is time not spent finding your new products, restocked items and updated prices.

There is a quality side too. Thousands of thin, near-identical pages make it harder for Google to pick the right URL for “women’s running shoes”. Internal link value also gets spread across URLs that should never rank.

Does Crawl Budget Actually Matter for Your Store?

For many small shops, no. Google’s crawl budget guidance is aimed at very large sites (roughly a million or more unique pages that change weekly) and medium-to-large sites (roughly 10,000 or more pages that change daily). Google calls those figures rough estimates.

The third group is the one most e-commerce managers fall into. These are sites with a large share of URLs sitting in “Discovered – currently not indexed” in Search Console. If that report is full of product and category URLs, I’d suspect filters and parameters first, whatever your catalogue size.

Crawl budget comes down to two things:

• Crawl capacity: how much Google can crawl without straining your server. Fast, stable responses raise it. 5xx errors and 429 responses lower it.
• Crawl demand: how much Google wants to crawl. This depends on the URLs it knows about, how popular they are and how often they change.

Faceted navigation hurts on both sides. It inflates the number of known URLs. And the extra requests slow a busy server, which can lower capacity.

Decide Which Facet Pages Deserve to Rank

Before I touch robots.txt, I sort the filters into two groups. Some filtered pages match real searches and deserve to be landing pages. Most do not.

A facet page is worth indexing when all of these are true:

• People search for that combination. Check your keyword tool and Search Console queries, for example “black leather boots” or “nike trail shoes”.
• It returns enough products to be useful, not one or two items.
• Its set of products is clearly different from the parent category.
• You are willing to give it a proper title, H1 and a short piece of unique intro text.

Here is how the common filter types usually work out:

My rule of thumb: let at most one or two facets stack on an indexable page. “Leather boots” can rank. “Brown leather boots size 9 under £100 sorted by newest” is a shopper’s filter, not a search landing page.

Your Faceted Navigation SEO Options, Compared

Google’s faceted navigation documentation offers two broad paths. Either stop filtered URLs from being crawled, or keep them crawlable and make them as efficient as possible. Most stores need a mix of both.

robots.txt Disallow for Low-Value Facets

This is the most reliable way to save crawling. It is also the method Google’s faceted navigation guidance leads with. Disallowed URLs are not fetched at all.

User-agent: *
# Block low-value filters wherever they appear in the query string
Disallow: /*?*size=
Disallow: /*?*price=
Disallow: /*?*rating=
Disallow: /*?*sort=
Disallow: /*?*view=
Disallow: /*?*instock=

The /*?*size= pattern matches the parameter whether it comes first or tenth in the query string. Test every pattern against real URLs from your site before you go live. One wildcard in the wrong place can block whole categories. I never skip this test.

Know the trade-offs:

• A blocked URL can still appear in the index, without a description, if enough links point to it. It is a crawl control, not an index control.
• Google cannot see a canonical tag or noindex on a blocked page. So do not rely on either for URLs you disallow.
• Google’s crawl budget guidance says not to use robots.txt to move budget around for a short time. Use it for URLs you never want crawled.

For more on how Google and other bots read the file, see my guide to robots.txt rules and AI crawlers.

URL Fragments for Filters

Google generally ignores everything after a # when crawling and indexing. Say your filters update the product grid with JavaScript and write the state to a fragment, such as /boots/#color=brown&size=9. Then there is only one URL to crawl.

The catch is that fragment-based filters can never rank on their own. Use them for filters you are sure have no search value. Keep your valuable facets on real, crawlable URLs.

rel=canonical to a Broader Page

A canonical on a filtered page tells Google which version you prefer. Point the low-value version at the page with fewer filters:

<!-- On /boots/?color=brown&sort=newest -->
<link rel="canonical" href="https://www.example.com/boots/?color=brown">

Canonicals combine signals. Over time, Google may crawl the non-canonical versions less often. But Google describes this as less effective than robots.txt for controlling crawl in the long run, because the URLs still have to be fetched for the canonical to be seen.

Canonicals are also hints, not orders. If the filtered page’s content is very different from the target, Google may ignore it. My guide to canonical tags and duplicate content covers when Google overrides your choice.

Noindex: Useful, but Not a Crawl Saver

A noindex robots meta tag keeps a filtered page out of search results. It does not stop crawling. Google has to fetch the page to see the tag. Its crawl budget guidance says clearly not to use noindex to manage crawl budget.

Use noindex when the problem is index bloat on a site where crawling is not the bottleneck. Or use it as a temporary step to get already-indexed filter pages dropped before you add a robots.txt block. If you block first, Google cannot see the noindex and the old URLs can hang around. I see this order mixed up quite often.

Nofollow on Filter Links

rel="nofollow" on filter links only helps if every link to that URL carries it. Google rates it as weaker than robots.txt over time. Treat it as a supporting step only.

Indexable Facet Pages, Built Properly

For the facets you do want ranking, treat them as category pages, not filter states:

1. Give them clean, stable URLs, ideally in the path: /boots/leather/, not /boots/?f=12.
2. Keep filter order fixed so /boots/leather/brown/ and /boots/brown/leather/ cannot both exist.
3. Write a unique title tag, H1 and a short intro that matches the search.
4. Use a self-referencing canonical.
5. Link to them from the parent category and from relevant product pages with normal <a href> links.
6. Return a real 404 when a combination has no products. Don’t serve an empty 200 page or redirect to a generic error page.

That last point comes straight from Google’s guidance. Empty or nonsensical filter combinations, duplicate filters and page numbers that don’t exist should return a 404 at the URL that was requested. Empty 200 pages are soft 404s (pages Google sees as empty even though they load), and Google keeps recrawling those.

Pagination Best Practice for Category Pages

Pagination is not the same problem as filtering. Page 2 of a category lists different products. So it is a unique page that Google needs to crawl to reach deeper products.

Google’s pagination guidance for e-commerce comes down to a few rules:

• Give each page a unique URL, such as ?page=2. Do not put page numbers in a fragment, because Google ignores fragments.
• Make each page self-canonical. Do not point pages 2, 3 and 4 at page 1. That tells Google to ignore the very pages that link to your deeper products.
• Link pages with crawlable links. Each page should link to the next with an <a href>. Google suggests linking back to the first page as well.
• Do not rely on rel=”next” and rel=”prev”. Google no longer uses them for indexing. Other search engines may, so they are harmless to keep.

<!-- On https://www.example.com/boots/?page=2 -->
<link rel="canonical" href="https://www.example.com/boots/?page=2">
...
<nav>
  <a href="/boots/">1</a>
  <a href="/boots/?page=3">Next</a>
</nav>

I’d add two e-commerce details. Page 1 should live at the clean category URL, so link to /boots/, not /boots/?page=1. And a request for ?page=40 when you only have 12 pages should return a 404.

Infinite scroll and “load more” buttons are fine for shoppers. Just make sure each batch of products also exists at a paginated URL that a crawler can reach without clicking or scrolling.

Parameter Handling Without the Old Search Console Tool

Search Console’s URL Parameters tool was retired in 2022. So you cannot tell Google how to treat parameters there any more. Parameter control now lives in your platform, your templates and robots.txt.

A clean parameter setup looks like this:

• Use the standard & separator and key=value pairs. Google’s guidance notes that commas, semicolons and brackets are often not recognised as separators.
• Enforce one parameter order. If a visitor creates ?size=9&color=brown, your server should change it to ?color=brown&size=9, ideally with a 301 redirect.
• Stop duplicates at the source. ?color=brown&color=brown should never be created, and should 404 or redirect if requested.
• Keep tracking parameters out of internal links. UTM tags and session IDs belong on outside campaign links only.
• Lowercase and trim values so ?color=Brown and ?color=brown do not split into two URLs.

# Before: three URLs for the same product set
/boots/?size=9&color=brown
/boots/?color=Brown&size=9
/boots/?color=brown&size=9&sessionid=8f2a1

# After: one canonical form, the others 301 to it
/boots/?color=brown&size=9

Your XML sitemap should list only canonical, indexable URLs: products, categories and the facet pages you chose to rank. My XML sitemap best practices guide covers how to split large catalogues into several files.

Internal Links: Control What Your Templates Expose

Googlebot finds filter URLs because your templates link to them. In my experience, changing what the filter sidebar outputs often works better than any directive.

• Valuable facets get real links. Show “Leather”, “Suede” and top brands as plain <a href> links to their clean landing page URLs.
• Low-value filters do not need crawlable links. Size, price sliders and sort menus can be form controls or buttons that update the grid with JavaScript, with the state in a fragment or a blocked parameter.
• Breadcrumbs point to canonical categories, never to a filtered state the visitor happened to arrive through.
• Product pages link back to the category, not to ?sort=popular&page=3.
• Add editorial links to key facet pages from buying guides and the parent category intro. That way your best facet pages are not orphaned once filter links stop being crawlable.

My internal linking strategy guide explains how to group these links into clusters that support your main category pages.

How to Audit Faceted Navigation With Logs and Crawl Stats

You cannot fix crawl waste you have not measured. I start with Search Console, then confirm with server logs.

Search Console Crawl Stats

Open Settings, then Crawl stats. The report shows total crawl requests, download size and average response time over roughly the last 90 days. It breaks these down by response code, file type, purpose (discovery or refresh) and Googlebot type.

Click into each breakdown and look at the example URLs. If the samples under “Discovery” and “OK (200)” are mostly parameter URLs, filters are eating your crawl. If average response time rises along with request counts, the extra crawling is putting load on your server.

Check the Page indexing report too. Large numbers of filter URLs under “Duplicate without user-selected canonical”, “Alternate page with proper canonical tag” or “Crawled – currently not indexed” point to the same problem.

Server Log Analysis

Crawl Stats shows samples. Logs show every request. Filter your access logs to verified Googlebot requests, then group them by URL pattern.

# Share of Googlebot hits that contain a query string
grep "Googlebot" access.log | awk '{print $7}' | grep -c "?"
grep "Googlebot" access.log | awk '{print $7}' | wc -l

# Most-crawled parameter names
grep "Googlebot" access.log | awk '{print $7}' | grep -o "[?&][a-z_]*=" 
  | sort | uniq -c | sort -rn | head -20

Anyone can fake the Googlebot user agent. So first verify a sample of IPs with reverse DNS (a lookup that confirms the IP really belongs to Google). My beginner’s guide to log file analysis walks through verification and the spreadsheet version of these checks.

The key comparison is crawl share against value. If sort and size parameters take 40% of Googlebot’s requests but bring no organic traffic, that is your priority. If new products take weeks to get their first Googlebot hit, crawl waste is costing you sales.

A Practical Audit Process

1. Export every parameter and path filter your platform can create, with an example URL for each.
2. Pull 30 days of logs and count verified Googlebot requests per parameter and per filter pattern.
3. Match those patterns against Search Console data to see which ones earn impressions and clicks.
4. Label each filter as index (real search demand), crawl-but-consolidate, or block.
5. Update templates, canonicals and robots.txt. Test every robots.txt pattern against real URLs first.
6. Re-run the log and Crawl Stats checks after four to six weeks. Compare the share of crawling that goes to products and indexable categories.

Mistakes That Make It Worse

These are the mistakes I see most often when filters are already causing trouble:

• Blocking and noindexing the same URLs. Google cannot see the noindex on a blocked page.
• Canonicalising all pagination to page 1. Deeper products lose their crawl path.
• Blocking a parameter your valuable facets use. If brand pages run on ?brand=, a blanket block removes them. Move them to clean paths first.
• Returning 200 for empty results. These become soft 404s that keep getting recrawled.
• Making facet pages indexable with no unique content. A filtered grid with a templated title rarely ranks, and it adds to your thin-page count.

Conclusion

Good faceted navigation SEO is about choosing. Pick the handful of filter pages with real search demand and build them as proper landing pages. Keep sort, size, price and stacked filters out of the crawl with robots.txt or non-crawlable controls. Give every paginated page its own canonical and a plain link to the next one.

Then measure. Your next step is to pull a month of server logs and run the parameter counts above. See how much of Googlebot’s attention goes to URLs you would never want to rank.