Crawl Budget Explained: When It Matters and How to Improve It

When Google is slow to pick up new or updated pages, people often blame crawl budget first. For most websites, it’s not the cause. Crawl budget is the number of URLs Googlebot can and wants to crawl on your site. It only becomes a real limit on large or fast-changing sites, or on sites that create far more URLs than they have real pages.

In this guide, I explain how Google decides how much to crawl, how to tell if your site has a real crawl problem, and what I’d fix first if it does. It follows Google’s own guide to managing crawl budget on large sites, turned into plain steps.

What Crawl Budget Actually Means

Google defines crawl budget as the set of URLs that Googlebot can and wants to crawl. Two separate forces decide it. The first is how much crawling your server can handle. The second is how much Google wants to crawl your content.

Budget is set per hostname. www.example.com and shop.example.com count as separate sites with separate crawl budgets. This matters if your blog, store and help centre live on different subdomains.

Crawling is not indexing. A crawled page still has to be judged worth indexing. So more crawling doesn’t guarantee more indexed pages, and crawl volume is not a ranking factor.

Crawl Capacity Limit

The crawl capacity limit is the ceiling on how hard Googlebot will hit your server. It covers how many connections Googlebot uses at once and the wait between fetches. Google sets it so it can crawl your important content without overloading your servers.

It goes up and down based on two things:

• Crawl health. If your site responds quickly and steadily, the limit rises. If it slows down or returns server errors, Googlebot backs off.
• Google’s own resources. Google has a lot of machines, but not unlimited ones. So it has to make choices across the whole web.

Search Console used to have a crawl rate setting that let you cap Googlebot by hand. Google removed it in January 2024. Today, the main lever you have over capacity is how well your server performs.

Crawl Demand

Crawl demand is how much Google wants to crawl your site. Even if your server could handle more, Googlebot won’t crawl more than it needs. Google names three main factors:

• Perceived inventory. Without guidance from you, Googlebot tries to crawl every URL it knows about. If many of those are duplicates, parameter versions or pages you don’t want crawled, they eat time that could go to your real content. This is the factor you control most.
• Popularity. URLs that are more popular on the internet tend to be crawled more often, to keep them fresh in the index.
• Staleness. Google recrawls pages often enough to pick up changes.

Site-wide events, such as moving to a new domain, can cause a short jump in demand.

Put the two together and you get the working definition. Crawl budget is what Googlebot can crawl (capacity) combined with what it wants to crawl (demand). If demand is low, Google crawls less, even when your server has room to spare.

Does Your Site Need to Worry About Crawl Budget?

Google’s guide is written for a certain kind of site. It gives rough thresholds, not hard rules:

• Large sites with about 1 million or more unique pages whose content changes fairly often, such as weekly.
• Medium or larger sites with about 10,000 or more unique pages whose content changes very fast, such as daily.
• Sites where a large share of URLs sit under Discovered – currently not indexed in Search Console’s Page Indexing report.

Say you run a 300-page service business site or a 2,000-product shop that updates a few items a week. Google will usually crawl everything it needs. For those sites, Google’s advice is simple: keep your sitemap current and check index coverage regularly. In my experience, most sites never need to think about crawl budget beyond that.

There is one important exception. A small site can still act like a large one if its templates create huge numbers of URLs. A shop with 2,000 products and five filters that can be combined freely can create hundreds of thousands of crawlable URLs. Then the problem is not the size of your catalogue. It’s the size of your URL inventory.

Signs of a Real Crawl Budget Problem

• New pages take weeks to appear in Google, even though they are linked and in your sitemap.
• Updated prices, stock levels or articles show old versions in search for a long time.
• The Page Indexing report shows a growing pile of Discovered – currently not indexed URLs.
• Your server logs show Googlebot spending most of its requests on parameter URLs, filters or old redirects instead of your key pages.
• The URL Inspection tool reports Hostload exceeded when you test a page.

What Wastes Crawl Budget

Most crawl budget problems are really inventory problems. Googlebot is busy fetching URLs that should never have been crawlable.

Faceted Navigation and URL Parameters

Faceted navigation means filters for colour, size, price, brand and sort order. They are useful for shoppers, but every combination can create a new URL. Add pagination and tracking parameters and the count grows fast. Googlebot can’t tell in advance that ?colour=red&size=10&sort=price_asc shows the same products as another URL. It has to fetch them to find out.

Google’s advice on faceted navigation is to block filter URLs you don’t need in search with robots.txt. Or you can use URL fragments (#) for filters, which Google generally ignores for crawling. For filter combinations with no results, return a 404, not an empty page. If some filtered pages do deserve to rank, such as a popular brand-plus-category page, make them clean, linked URLs. Keep the rest out of the crawl. On shops, this is the area I check first, because it’s where I usually find the most waste.

Duplicate Content

When the same page can be reached through several URLs, Googlebot’s attention gets split. Common causes are HTTP and HTTPS versions, www and non-www, trailing slashes, printer-friendly versions, and products that sit under several category paths.

Consolidate duplicates so one URL stands for each piece of content. Redirects, consistent internal linking and canonical tags (a line of code that tells Google which version is the main one) all help. There is more on how Google picks between versions in my guide to how Google handles duplicate content. Keep in mind that a canonical tag is a signal, not a crawl block. Google may still fetch the duplicate URLs, just less often over time.

Soft 404s

A soft 404 is a page that tells users it doesn’t exist but returns a 200 (OK) status code. Think of an “out of stock, no longer available” product page, or an empty search results page served as a normal page. Google keeps recrawling these because the server says they are fine.

Return a real 404 or 410 for pages that are gone for good. Google’s Page Indexing report lists the soft 404s it has found, which gives you a ready-made list to work through.

Redirect Chains

Each hop in a redirect chain is a separate request. Picture a link that goes from an old HTTP URL to HTTPS, then to www, then to a new slug. That costs three fetches before Googlebot reaches any content.

Update internal links to point straight at the final URL. Then collapse chains so each old URL redirects in one hop. My guide to choosing between 301 and 302 redirects covers which status code to use. I usually see long chains on sites that have been through several redesigns or HTTPS moves.

Infinite Spaces

Some features create an endless number of URLs. Examples are calendars with a never-ending “next month” link, crawlable internal search results, relative links that keep adding path segments, and session IDs added to every link.

Find the pattern, stop linking to it where you can, and disallow it in robots.txt. For calendars, limit the range of dates that link onward.

Low-Value and Outdated Pages

Thin tag archives, expired listings and old campaign pages all compete for crawl time. Remove pages with no purpose with a 404 or 410, and merge thin pages where it makes sense.

Robots.txt vs Noindex for Crawl Control

Mixing these two up is one of the most common mistakes in crawl budget advice. Robots.txt and noindex do different jobs.

Google is clear on this. Don’t use noindex to manage crawl budget, because Google still requests the page and only then drops it. Use robots.txt for URLs you don’t want crawled at all. Use noindex when the goal is keeping a page out of the index.

Don’t use both on the same URL either. If robots.txt blocks a page, Googlebot never sees its noindex tag.

Google’s guide gives two more robots.txt warnings:

• Don’t use it to move budget around for a short time. Blocking a section for a week won’t push that crawling onto other pages, unless Google is already hitting your server’s limit.
• Don’t use it for removed pages. Blocked URLs stay in Google’s crawl queue much longer, and they get recrawled when you unblock them. Return 404 or 410 instead.

Also note that Googlebot ignores the crawl-delay rule in robots.txt. If you need Google to slow down, your server’s responses are what count.

A typical crawl-focused robots.txt for a shop might look like this:

User-agent: *
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /search
Disallow: /*?sessionid=

Sitemap: https://www.example.com/sitemap.xml

Test wildcard rules against real URLs before you put them live. A pattern that is too broad can block product pages. I never ship a new robots.txt rule without testing it first. My robots.txt guide covers the syntax and the mistakes I see most.

Server Speed and Crawl Capacity

The capacity limit follows crawl health. So a faster, more stable server lets Googlebot fetch more URLs in the same time. Google’s guide points to two practical improvements:

• Faster responses. Cut server response time with caching, efficient database queries and enough hosting capacity. Lighter pages with fewer heavy files also help, because Googlebot fetches CSS, JavaScript and other files when it renders pages.
• HTTP caching. Support conditional requests with ETag or Last-Modified headers. When a page hasn’t changed, your server can return 304 Not Modified with no body. This saves bandwidth for both sides.

Server errors work the other way. A run of 5xx responses or timeouts tells Googlebot to back off. If your server is truly overloaded and you need Google to crawl less in an emergency, returning 503 or 429 for a short time works. But Google warns against doing this for more than two or three days. A longer period can make Google crawl your site less often over the long term.

If the URL Inspection tool reports Hostload exceeded, Google wanted to crawl more but held back because of your server. That is the clearest sign that more server resources will lead to more crawling.

Sitemaps and lastmod

An XML sitemap doesn’t raise your crawl budget. But it helps Google spend the budget it has on the right URLs. Keep these points in mind:

• List only canonical, indexable URLs that return 200. Leave out redirects, noindexed pages, blocked URLs and parameter versions.
• Use <lastmod> and keep it accurate. It should show the last real change to the page content, not the time the sitemap was generated. Google uses lastmod when it is always accurate and ignores it when it isn’t.
• Split large sites into several sitemaps by section, such as products, categories and articles. Then you can see which areas are indexed well.
• Google ignores <priority> and <changefreq>, so don’t spend time tuning them.

Internal linking still matters. Pages linked clearly from navigation and category pages are found and refreshed more reliably than pages that only exist in a sitemap.

How to Use the Crawl Stats Report

The Crawl Stats report in Search Console shows Google’s crawling of your site from Google’s side. You’ll find it under Settings, then Crawl stats. It’s only available for domain properties or URL-prefix properties at the root of a host. Google calls it a report for advanced users and says sites with fewer than about a thousand pages usually don’t need it. If you are new to Search Console, start with my walkthrough of the Search Console reports that matter.

Google’s Crawl Stats report documentation explains each chart. These are the parts I check:

• Total crawl requests, download size and average response time. Look for sudden drops in requests along with rising response times. That pattern usually means your server is slowing Googlebot down.
• Host status. This flags problems with fetching robots.txt, DNS and server connections over the past 90 days. Any red status needs attention first, because failed robots.txt fetches can make Google pause crawling.
• By response. A high share of 404s, 5xx errors or redirects (301, 302) shows where requests are spent on URLs that don’t serve content.
• By file type. If JavaScript, JSON or “other” types lead, heavy files or API calls may be using a big slice of crawling.
• By purpose. Discovery means URLs Google hadn’t crawled before. Refresh means recrawls of known URLs. A site publishing lots of new content should see steady discovery activity.

Click into any row to see example URLs. They are samples, not a full list. But they quickly show patterns, such as a forgotten parameter. For the full picture, compare them with your server logs. When I have access to logs, I always use them.

How to Improve Crawl Budget on a Large Site

If your site meets Google’s thresholds or shows the warning signs above, work through these steps in order.

1. Check if you have a problem. Review the Page Indexing report for Discovered – currently not indexed URLs and the Crawl Stats report for host status issues. Test a few slow-to-index pages in URL Inspection. On a big site, the URL Inspection API lets you check many URLs at once.
2. Map where Googlebot spends its time. Use the Crawl Stats example URLs and your server logs to group crawled URLs by pattern: products, categories, filters, parameters, redirects and errors.
3. Cut the URL inventory. Block filter, sort, internal search and infinite-space patterns in robots.txt. Consolidate duplicates with redirects and canonicals. Return 404 or 410 for removed pages and empty filter results.
4. Fix soft 404s and redirect chains. Make gone pages return proper status codes. Point internal links at final URLs and collapse chains to a single hop.
5. Speed up the server. Lower response times, support 304 Not Modified responses and fix repeated 5xx errors or timeouts.
6. Clean up your sitemaps. Include only canonical, indexable URLs and keep lastmod accurate.
7. Monitor for a few weeks. Watch Crawl Stats and Page Indexing for a shift toward more requests on your important URLs and fewer on waste. Crawl patterns take time to change, so judge trends, not single days.

Conclusion

Crawl budget is a real limit for large and fast-changing sites, and a distraction for most others. Google crawls what your server can handle and what it thinks is worth crawling. So the fixes come down to two things: shrink the pile of useless URLs, and keep your server fast and stable.

Start by checking the Page Indexing and Crawl Stats reports. If you find crawl waste, the biggest wins usually come from faceted URLs and duplicates. So my suggested next step is to review how your site handles duplicate content before you touch anything else.