You opened Search Console, found the Crawl Stats report, and now you are looking at charts full of requests, bytes and milliseconds. What do they mean, and should you worry? The crawl stats report shows how often Google crawls your site, what it asks for, how your server answers and whether your site was available when Googlebot came by.
For most small sites, this report is a health check, not a daily task. For larger sites, it is one of the best early warnings you get from Google. Below I explain every part of the report in plain words, what normal looks like, and the patterns I treat as a real problem.
What the Crawl Stats Report Shows
The Crawl Stats report is Google’s record of its crawling on your site over roughly the last 90 days. Crawling is when Googlebot (Google’s crawler) requests a URL from your server to read it. The report covers:
- Totals over time: total crawl requests, total download size and average response time.
- Host status: whether Google had trouble reaching your site because of robots.txt, DNS or server problems.
- Breakdowns: crawl requests grouped by response code, file type, purpose and Googlebot type.
Click any row in a breakdown and you get a detail page with example URLs. Google says these examples are a sample, not a full list. So use them to spot patterns, not to count every broken URL.
Which Sites Really Need It
Google describes this as a report for advanced users and says sites with fewer than about 1,000 pages usually don’t need it. If you run a 40-page service business site, a quick look every few months is enough.
It matters much more if your site has thousands of pages, a lot of new content each day, an online shop with filters, or a recent migration. Those are the sites where crawling problems turn into indexing problems.
How to Find the Crawl Stats Report in Search Console
The report is not in the main left menu, which is why many people never find it. Here is the path:
- Open Google Search Console and pick your property.
- Click Settings near the bottom of the left menu.
- Under Crawling, find Crawl stats and click Open report.
Why the Report Might Be Missing
The report only works for root-level properties. That means a Domain property such as example.com, or a URL-prefix property at the root such as https://example.com/. If your property is a folder, like https://example.com/blog/, you won’t see it.
The fix is to add a Domain property, which you verify with a DNS record. I set up a Domain property on every site I work on anyway, because it also groups all subdomains and both http and https in one place.
How Google Counts the Numbers
Before you read any chart, it helps to know how Google counts. The rules below come from Google’s Crawl Stats report help page. This explains most of the confusion I see when people compare the report with other tools.
- Exact URLs, not canonicals. The report shows the exact URL Google requested. It does not merge duplicates into the main (canonical) version.
- Each redirect hop counts. If a URL redirects twice before landing, that is three requests. Redirects are listed under “Other file type”.
- Failed attempts count too. DNS failures, connection failures and fetches Google skipped because robots.txt was unavailable are all included in the total.
- Only your own host. Images or scripts loaded from another domain, such as a CDN on a different hostname, are not counted here.
- Subdomains roll up. On a Domain property you see requests to subdomains, but one subdomain’s report won’t show its sibling subdomains.
Google also notes the totals can differ a little from your server logs. A small gap is normal. A big gap is usually caused by a long robots.txt outage, where Google counts crawls it wanted to make but didn’t.
Reading the Three Main Charts
At the top of the report you get one chart with three metrics. Click each box to switch it on or off. I look at them together, because one number on its own rarely tells the full story.
Total Crawl Requests
This is how many requests Google made to your site each day. It includes successful and failed requests, and repeat requests for the same URL.
Do not judge this number against other sites. A news site and a local plumber will have very different numbers, and both can be healthy. What matters is your own trend. A steady line with normal ups and downs is fine. A sharp, lasting change is what you want to explain.
Total Download Size
This is how many bytes Googlebot downloaded while crawling. Google only fetches a cached resource, like a CSS file it already has, once.
If download size jumps while crawl requests stay flat, your pages got heavier. I usually see this after a theme change, a new page builder or uncompressed images.
Average Response Time
This is the average time your server took to respond to Googlebot, for every resource it fetched. Each image, script and stylesheet counts on its own.
Google does not publish a target number for this metric, so I don’t chase one. I watch the trend. If response time climbs and crawl requests drop at the same time, your server is likely slowing Google down. Google’s guide to managing crawl budget explains that Google crawls less when a site responds slowly or returns server errors, and more when it stays fast and stable.
One thing to keep clear: this is server response time for Googlebot, not how fast your page feels to visitors. For real user speed, look at your Core Web Vitals instead.
Host Status: robots.txt, DNS and Server Connectivity
Host status tells you whether Google could reach your site at all. It shows one of three results:
- Green tick: no major availability issues in the last 90 days. Nothing to do.
- Issues more than a week ago: there were problems, but not recently. Check whether they were a one-off.
- Issues in the last week: something recent went wrong. Look into it now.
Click the host status box to see three charts. Each one has a dotted red line that marks an unusual failure rate. Spikes above that line are what you care about.
robots.txt Fetching
Before crawling, Google requests your robots.txt file (the file that tells crawlers which parts of your site to skip). Google treats these responses as successful:
- A 200 response, even if the file is empty or has errors.
- A 403, 404 or 410, which Google treats as “no robots.txt file”, so it crawls freely.
A 429 or any 5XX error counts as a failure, and this is where it gets serious. According to Google, after a failed robots.txt fetch it stops crawling for the first 12 hours. From 12 hours up to 30 days, it uses the last good copy of the file. After 30 days, it crawls without limits if your homepage loads, or stops if it doesn’t.
In my experience, robots.txt failures usually come from a firewall, a security plugin or a CDN rule that treats Googlebot as a bot to block. If this chart shows errors, it’s the first thing I fix. The Search Console robots.txt report shows each file Google fetched, its status and any errors. If you want to check your rules too, see my robots.txt guide with examples and common mistakes.
DNS Resolution
DNS is the system that turns your domain name into a server address. This chart shows when your DNS server didn’t recognise your hostname or didn’t answer.
Small blips happen. If failures stay high, talk to your DNS provider or domain registrar. I also double-check DNS records after any hosting or email change, because that’s when they get edited by mistake.
Server Connectivity
This chart shows when your server didn’t respond or sent back an incomplete response. Common causes are an overloaded shared host, a server that goes down at peak times, or a firewall cutting off Googlebot’s connections.
If you see spikes, compare the dates with your hosting provider’s uptime records and any traffic peaks. Repeated connectivity errors lead Google to crawl less, so it’s worth sorting out even if visitors don’t seem to notice.
The Four Crawl Request Breakdowns
Below the charts, Google splits your crawl requests four ways. Each one answers a different question.
By Response
This shows the HTTP status codes your server returned. The percentages are based on requests, not unique URLs. Google groups them like this:
| Group | Responses included | What I do |
|---|---|---|
| Good | 200 OK, 301 and 308 (permanent redirect), 302 and 307 (temporary redirect), Moved (other) such as a meta refresh, 304 Not modified | Nothing, unless redirects make up a large share of requests |
| Possibly good | 404 Not found | Check the examples. Fix only 404s for pages that should exist or that have useful links |
| Bad | robots.txt not available, 401 and 407 (unauthorised), 5XX server errors, other 4XX, DNS errors, fetch errors, page could not be reached, timeouts, redirect errors, other errors | Investigate, starting with 5XX errors and timeouts |
A healthy site is mostly 200 responses. Google itself says fixing every 404 is not always needed or even a good idea. Deleted products and old URLs should return 404 or 410.
Two things I always check here. First, a high share of 301s usually means internal links still point to old URLs, so Google keeps following redirects. Second, any steady level of 5XX errors needs attention, because server errors are a direct signal to Google to slow down.
By File Type
This shows what kind of files Google fetched: HTML, images, JavaScript, CSS, JSON, PDF, video, syndication feeds and more. “Other file type” includes redirects, and “Unknown (Failed)” means the request failed before Google could tell the type.
On a typical site, HTML, images, JavaScript and CSS make up most of the requests. If JSON or “Other” is unusually large, open the examples. It often points to API calls, tracking scripts or redirect chains that Google keeps hitting.
By Purpose
Google splits requests into two purposes:
- Discovery: URLs Google has never crawled before.
- Refresh: recrawls of pages Google already knows.
Most established sites are mostly refresh, and that’s normal. If you publish a lot of new content and discovery stays very low, Google may not be finding your new pages quickly. Check that they are in your XML sitemap and linked from other pages on your site.
If discovery suddenly shoots up and you didn’t add new content, be suspicious. It often means Google found a pile of parameter or filter URLs, or a bug is creating endless new links.
By Googlebot Type
This shows which Google crawler made the requests. The main types are:
- Smartphone and Desktop: Googlebot crawling as a mobile or desktop browser. Smartphone is usually the largest share, because Google uses mobile-first indexing.
- Image and Video: crawlers fetching media files.
- Page resource load: extra fetches for images, CSS and scripts that Google needs to render (build) a page.
- AdsBot: checks pages used in Google Ads. Google says spikes are often caused by adding many Dynamic Search Ads targets.
- StoreBot: the crawler for shopping-related pages.
- Other agent type: other Google crawlers not listed above.
Keep in mind that all Google crawlers share the same crawl capacity for your site. If AdsBot is suddenly very busy, there may be less room for Googlebot itself.
Common Patterns and What They Usually Mean
Once you know what each part shows, the real value is in reading changes. These are the patterns I look for.
A Sudden Drop in Crawling
Google lists a few likely causes for a crawl drop:
- A new robots.txt rule, or an old one that now blocks more than you meant.
- Slower responses, which make Google crawl more gently.
- More server errors in the response breakdown.
- Content that rarely changes, or that Google sees as lower quality.
A drop is not a penalty. It’s Google reacting to what it sees. I start with host status and the response breakdown, because those causes are the quickest to confirm and fix. I also check whether Google reported a crawling problem on those dates, using the Google Search Status Dashboard.
A Sudden Spike in Crawling
A spike usually follows something you changed: unblocking a big section, adding a large new section, or adding many Dynamic Search Ads targets. If you can’t link it to a change, open the purpose and file type breakdowns. Faceted navigation (filters for size, colour, price and so on) is the usual source of a surprise spike on shop sites.
If the extra crawling is hurting your server, Google suggests a short-term fix: return 503 or 429 responses, or block parts of the site in robots.txt. Google says not to keep this up for more than two or three days, or it can hurt how your pages are crawled and indexed. Google adjusts crawl rate on its own over time, and you can also ask Google for a lower rate if the problem continues.
Crawl Rate Seems Too Low
You can’t ask Google to crawl more. What you can do is keep your sitemap up to date, make sure important pages aren’t blocked, link to new pages from strong pages, and keep your server fast. Over time, Google crawls more when it sees useful, fresh content and a stable server.
How I Review the Crawl Stats Report
On larger sites I check the crawl stats report monthly, and right after any migration, redesign or hosting change. Here is the order I follow:
- Host status first. If it isn’t green, open the failing chart and match the dates with any changes on your site or hosting.
- Look at the three main charts together. Note any lasting change in requests, download size or response time, and when it started.
- Check the response breakdown. Look for 5XX errors, a rising share of 404s and a high share of redirects.
- Check purpose and file type. Make sure discovery fits how much new content you publish, and that odd file types aren’t eating requests.
- Open the example URLs. Look for patterns like parameters, old folders or redirect chains.
- Confirm single URLs. For a specific page, use the URL Inspection tool to see its crawl and index status in detail.
- Fix, then watch. Make the change and add a note in your own records with the date, so you can match it to the charts later.
What This Report Can’t Tell You
The Crawl Stats report is useful, but it has limits. I don’t use it for these jobs:
- Finding every broken URL. The examples are a sample. Use a site crawler or your server logs for a full list.
- Checking indexing. Crawled does not mean indexed. The Page indexing report shows which pages are in Google.
- Measuring page speed. Response time here is for Googlebot fetches. It is not a Core Web Vitals score.
- Checking non-Google bots. Bingbot and other crawlers only show up in your server logs.
If you are not sure which Google tool answers which question, my guide to free Google tools for SEO explains what each one is for.
Conclusion
The crawl stats report is Google telling you how it experiences your server. Start with host status, compare your trend lines with your own past rather than with other sites, and use the breakdowns to find the cause of any sudden change.
Your next step: open Settings in Search Console, check that host status is green, and look at the share of 5XX and redirect responses. If robots.txt fetching shows any failures, fix that first, because it can stop Google crawling your whole site.
