If you used to check your robots.txt file with Google’s old robots.txt Tester, that tool is on its way out. On 15 November 2023, Google added a new robots.txt report to Search Console and said it is sunsetting the tester. The new report shows which robots.txt files Google found on your site, when it last crawled them, and any errors or warnings it hit.
In this article I explain what the robots.txt report shows, how to open it, how to read each column and status, and what to use now that you can no longer test URLs in the old tool.
What Changed on 15 November
Google made three related changes in one announcement:
- A new robots.txt report inside Search Console, found in the Settings area.
- More robots.txt information in the Page indexing report, so you can see which pages are affected by your rules.
- The robots.txt Tester is being retired. I would stop relying on it now.
A robots.txt file is a plain text file at the root of your site, such as https://example.com/robots.txt. It tells crawlers like Googlebot which parts of the site they may and may not crawl.
The old tester was a leftover from the previous version of Search Console. It only worked with URL-prefix properties, and it looked and felt like a separate tool. The new report sits inside the current Search Console and works with Domain properties, which is how most people set up sites today.
What the robots.txt Report Shows
The robots.txt report answers a simple question: which robots.txt files does Google see for my site, and did it read them without problems?
According to Google’s Search Console help page for the robots.txt report, it shows:
- The robots.txt files Google found for the top 20 hosts on your site.
- The last time Google tried to crawl each file.
- Any warnings or errors Google found when it read the file.
- The versions Google fetched over the last 30 days.
- A way to ask Google to recrawl a file.
A “host” is one version of your site name, such as www.example.com, shop.example.com or blog.example.com. Each host needs its own robots.txt file, because the rules in one file never apply to another subdomain.
Which Properties Can Use It
The report only works at the domain level. That means:
- A Domain property, such as
example.com. - A URL-prefix property with no path, such as
https://www.example.com/.
If your property is a folder, like https://www.example.com/blog/, you won’t see the report. That makes sense, because robots.txt lives at the root of the host, not inside a folder.
In a Domain property, Google picks the top 20 hosts by crawl rate. For each host it checks both the http:// and https:// versions, so the table can show up to 40 rows.
How to Open the robots.txt Report
You won’t find the report in the main left-hand menu with Pages and Performance. It sits in Settings.
- Open Search Console and pick a Domain property or a root-level URL-prefix property.
- Click Settings near the bottom of the left menu.
- In the Crawling section, find the robots.txt row and click Open report.
- Look at the table of files. Each row is one robots.txt URL Google checked.
- Click a row to see the last version Google fetched, with any issues highlighted.
When I set up Search Console for a new site, this is now one of the first places I look after the Page indexing report. It takes less than a minute and tells me straight away if Google can’t read the file.
How to Read Each Column
The table has five main columns. Here is what each one tells you.
| Column | What it means | What I look for |
|---|---|---|
| File | The full robots.txt URL Google checked | Any host I didn’t expect, like an old staging subdomain |
| Fetch status | The result of Google’s latest attempt to get the file | Anything other than “Fetched” on a live host |
| Checked on | When Google last tried to crawl the file, in your local time | A date that matches when I expect Google to have seen my latest change |
| Size | The file size in bytes; empty if the last fetch failed | A very large file, or a size of zero when I expect rules |
| Issues | The number of parsing errors and warnings | Any number above zero |
A file only appears in the table if Google tried to fetch it in the last 30 days. So if a host you care about is missing, Google may not be crawling it much, or it may not be one of the top 20 hosts.
Errors vs Warnings
Google reads your file line by line. If a line has a problem, Google ignores that line and still uses the rest of the file.
- An error means a rule could not be used at all.
- A warning means something looked odd, but it did not stop the rule from working.
Click a file to open the fetched version. Problem lines are highlighted, and you can jump between them. Common causes I see are typos in field names, rules placed before any User-agent line, and stray text pasted in from a CMS or a plugin.
Fetch Status Values Explained
The fetch status is the most important column. There are three values you will meet.
Fetched
Google got the file and read it. This is what you want for every live host. Even with “Fetched”, check the Issues column, because some lines may have been ignored.
Not Fetched – Not Found (404)
The file does not exist at that URL. For Google, no robots.txt file means it may crawl every URL on that host. That is fine for many small sites. It is a problem if you thought you were blocking something.
After 30 days with this status, the row drops out of the report. Google still keeps checking for the file in the background.
Not Fetched – Any Other Reason
Something else went wrong, such as a server error, a timeout or a DNS problem. This is the status I treat as urgent.
Google’s robots.txt specification explains why. If the robots.txt URL returns a server error (a 5xx code), Google stops crawling the whole site for the first 12 hours while it keeps retrying the file. After that, for up to 30 days, it uses the last good copy it had. So one broken robots.txt file can slow crawling of a perfectly healthy site.
A 4xx response other than 429 is different. Google treats it as if there is no robots.txt file and crawls without limits. That means a 403 or 401 on robots.txt does not protect anything, and it is not a way to slow Google down.
Versions and the 30-Day History
Each file has a Versions view. It lists the fetches from the last 30 days. Google only records a new version when the file content or the fetch result changes, so you don’t get 30 identical copies.
I find this the most useful part of the robots.txt report. If crawling dropped on a certain date, I check whether the file changed around that day. It is a quick way to catch a developer, a plugin or a site migration that quietly rewrote the file.
Keep in mind that the history only goes back 30 days. If you want a longer record, keep your robots.txt file in version control or save a dated copy whenever you change it.
How to Request a Recrawl of robots.txt
Google usually caches robots.txt for up to 24 hours. Most of the time, it will pick up your change within a day without any help. But if you fixed something serious, you can ask Google to fetch the file sooner.
- Fix the live file on your server first, and load it in your browser to confirm the change is there.
- Open the robots.txt report in Search Console.
- Find the row for the host you changed.
- Click the three-dot menu at the end of the row and choose Request a recrawl.
- Come back later and check that “Checked on” shows a new time and the status is “Fetched”.
I only use this in two cases. First, when I have removed a rule that was blocking important pages by mistake. Second, when the file was returning an error and I have fixed it. For a small tidy-up, I let Google find the change on its own.
A recrawl of robots.txt does not mean Google will recrawl the pages you just unblocked straight away. It only updates the rules. Google will get to those pages in its normal crawl. For a handful of key pages, you can also request indexing in the URL Inspection tool.
What Replaces the robots.txt Tester
This is the part most people will miss. The old tester let you paste a URL, pick a Googlebot type and see which line allowed or blocked it. You could also edit the rules in the browser and test them before going live. The new report does not do either.
Google has not launched a direct replacement for that test-a-URL feature. Here is what I use instead.
URL Inspection for Live Pages
The URL Inspection tool tells you if a specific page is blocked by robots.txt. Run a live test, and it shows whether crawling is allowed. It won’t show the exact line that blocks the page, but it answers the yes-or-no question for the URL Google cares about.
If you check many URLs, the URL Inspection API lets you run the same check in bulk with a script or a tool that supports it.
The Page Indexing Report for Patterns
Google said it now shows relevant robots.txt information in the Page indexing report. Two statuses matter here:
- Blocked by robots.txt: Google did not crawl these pages because of your rules.
- Indexed, though blocked by robots.txt: Google indexed the URL anyway, usually because other pages link to it, without crawling its content.
I check the example URLs under both statuses. If I see product pages, blog posts or service pages in “Blocked by robots.txt”, something is wrong with a rule.
A Third-Party robots.txt Checker for Testing New Rules
For testing rules before they go live, you now need a separate tool. Many SEO crawlers and free online checkers let you paste a robots.txt file and test URLs against it. Google has also released its own robots.txt parser as open source code, which some tools are built on.
My rule: test any new or edited rule against a short list of real URLs before you upload it. Include pages you want blocked and pages you must keep open, like your home page, key categories and your CSS and JavaScript files.
robots.txt Rules That Commonly Trip People Up
The report tells you if Google could read your file. It does not tell you if the rules do what you meant. These are the mistakes I see most often.
- Blocking the whole site by accident. A single
Disallow: /underUser-agent: *blocks everything. This often comes over from a staging site during a launch. - Using robots.txt to hide pages from search. Robots.txt stops crawling, not indexing. A blocked page can still show in results as a bare URL. Use a
noindextag on a crawlable page instead. For urgent cases, the Search Console Removals tool can hide a URL for a while. - Blocking CSS and JavaScript. Google needs these files to render your pages. Blocking them can make pages look broken to Google.
- Forgetting that paths are case-sensitive.
Disallow: /Private/does not block/private/. - Expecting one file to cover subdomains.
www.example.com/robots.txtdoes nothing forshop.example.com.
How Google Picks Which Rule Wins
When an Allow and a Disallow rule both match a URL, Google uses the most specific rule, meaning the one with the longest matching path. If both are equally specific, the less restrictive Allow rule wins.
Here is a simple example:
User-agent: *
Disallow: /shop/
Allow: /shop/sale/Google can crawl /shop/sale/winter-coats because /shop/sale/ is the longer match. It cannot crawl /shop/basket, which only matches /shop/.
Two more details from Google’s specification are worth knowing. Google only reads the first 500 KiB of a robots.txt file and ignores anything after that. And Google follows only the group with the most specific matching user agent, so a Googlebot group replaces your User-agent: * rules for Googlebot rather than adding to them.
That same group logic applies to the new AI tokens. If you added a Google-Extended group to block Bard and Vertex AI training, that group is separate from your Googlebot rules. The robots.txt report is a good place to confirm Google fetched the updated file without errors.
My Routine for Checking the robots.txt Report
You don’t need to watch this report every day. Here is how I fit it into normal work:
- Monthly: a quick look to confirm every live host shows “Fetched” with zero issues.
- After any robots.txt edit: check the next day that “Checked on” has updated and the new version appears.
- After a site launch, redesign or migration: check on the day, because this is when staging rules and blanket blocks slip through.
- When crawling or traffic drops suddenly: open the Versions history before anything else and see if the file changed.
- When a new subdomain goes live: confirm it has its own robots.txt file and that the file returns a normal 200 response.
In my experience, most small business sites have a short robots.txt file that rarely changes. For them, the main value of the report is catching server errors. For larger sites with many subdomains, the host list alone is useful, because it often shows old hosts nobody remembered.
Is the New robots.txt Report Better Than the Old Tester?
It depends on what you used the tester for.
For monitoring, the new report is clearly better. It covers many hosts at once, works with Domain properties, keeps 30 days of versions and shows the fetch status that actually affects crawling.
For testing, it is a step back. Losing the ability to test a single URL against your rules inside Search Console is a real gap, and I expect many SEOs will miss it. My honest view is that you should treat the report as your health check and keep a separate tool for testing rule changes.
Final Thoughts on the Search Console robots.txt Report
The robots.txt report is now the official place to see what Google thinks of your robots.txt file. It shows every file Google found for your main hosts, whether it could fetch them, which lines have problems and how the file changed over 30 days.
Your next step is simple. Open Settings in Search Console, find the robots.txt report and confirm every live host shows “Fetched” with no issues. Then check the “Blocked by robots.txt” examples in the Page indexing report to make sure your rules only block what you meant to block.
