Robots.txt Guide: Rules, Examples and Common Mistakes

Maybe Search Console says a page is “Blocked by robots.txt”. Or you have a robots.txt file and are not sure what it does. A robots.txt guide should answer one thing first: robots.txt is a small text file that tells search engine crawlers which parts of your site they may visit. It controls crawling, not indexing. That one difference causes most of the mistakes I see.

In this guide I explain how the file works, what each rule means, ready-to-use examples, how Google handles conflicts and errors, and the mistakes that quietly hurt rankings.

What robots.txt Is and What It Does

Robots.txt is a plain text file that sits at the root of your site, like https://example.com/robots.txt. Crawlers (programs that visit pages, like Googlebot and Bingbot) check it before they crawl. It follows the Robots Exclusion Protocol, which became an official internet standard, RFC 9309, in 2022.

Google says the main job of robots.txt is to manage crawler traffic. You use it to stop bots wasting time on pages that do not need crawling, or to keep your server from being overloaded.

Here is what it is good for:

• Keeping crawlers out of internal search results, cart pages and endless filter combinations.
• Stopping images, videos or PDFs from showing in Google results.
• Pointing crawlers to your XML sitemap.
• Saying no to specific bots, like some AI crawlers.

And here is what it cannot do:

• It cannot hide a page from Google. A blocked URL can still be indexed if other pages link to it. Google just shows it without a description.
• It cannot protect private content. The file is public. Anyone can open it and see the paths you listed.
• It cannot force bad bots to behave. Following the file is voluntary. Googlebot and other well-known crawlers respect it. Scrapers often do not.

In my experience, most small sites need only a few lines in robots.txt. The file does more harm when people add rules “just in case” than when they leave it simple.

Robots.txt Rules Explained

Each line in the file is a field: value pair. Google supports four fields. Anything else is ignored.

User-Agent

The user-agent line says which crawler the rules below it apply to. User-agent: * means all crawlers. User-agent: Googlebot means only Google’s main crawler. A user-agent line plus the rules under it is called a group.

The user-agent name is not case-sensitive, so googlebot and Googlebot both work.

Disallow

Disallow tells the crawler not to visit URLs that start with the path you give. Disallow: /admin/ blocks everything inside the admin folder. An empty Disallow: blocks nothing.

Allow

Allow makes an exception inside a blocked area. It is useful when you block a folder but want one file inside it crawled. Google and Bing both support it.

Sitemap

The sitemap line gives the full URL of your XML sitemap. It must be an absolute URL, including https://. You can list more than one sitemap, and the line does not belong to any user-agent group, so you can place it anywhere in the file.

Wildcards: * and $

Google and Bing support two special characters in paths:

• * matches any run of characters. Disallow: /*?sort= blocks any URL that contains ?sort=.
• $ marks the end of the URL. Disallow: /*.pdf$ blocks URLs that end in .pdf.

Paths are case-sensitive. Disallow: /Blog/ does not block /blog/. Lines that start with # are comments, which crawlers ignore. I use comments to note why a rule exists, so the next person does not delete it by guesswork.

Robots.txt Examples You Can Copy

These examples cover the most common needs. Change the paths to match your own site.

Allow Everything

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

This is a fine starting file for most small business sites. Having no robots.txt at all also allows everything, but I like having the file so the sitemap line is there.

Block a Folder but Allow One File Inside It

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This is the default pattern on many WordPress sites. The admin area is blocked, but the file that some themes and plugins need for front-end features stays open.

Block Internal Search and Filter URLs

User-agent: *
Disallow: /search/
Disallow: /*?s=
Disallow: /*?filter=
Disallow: /*&sort=

Internal search pages and filter combinations can create thousands of near-duplicate URLs. Blocking them keeps crawlers focused on the pages you actually want found. On online stores, this is usually the most useful thing robots.txt does.

Block One File Type

User-agent: *
Disallow: /*.pdf$

Block One Crawler Completely

User-agent: GPTBot
Disallow: /

This blocks OpenAI’s training crawler but leaves every other bot alone. I cover AI crawlers in more detail below.

Block a Whole Site (Staging Only)

User-agent: *
Disallow: /

This blocks every compliant crawler from every page. Use it only on a test or staging site, and even there a password is the better choice.

How Google Reads Your robots.txt

This is the part most guides skip. Knowing how Google handles the file saves you from rules that do the opposite of what you expect.

Only One Group Applies to Each Crawler

A crawler picks the single group whose user-agent matches it most closely. It ignores the rest. If you have a User-agent: * group and a User-agent: Googlebot group, Googlebot follows only the Googlebot group.

So if you write a special group for one bot, repeat every rule it still needs. I often see sites add a Googlebot group with one line and accidentally open up everything they had blocked under *.

Also note that Google’s AdsBot crawlers ignore the * group. If you want to block them, you have to name them directly.

The Most Specific Rule Wins

When an Allow and a Disallow both match a URL, Google uses the rule with the longest matching path. If they are equally long, Google uses the less restrictive one, which is Allow.

User-agent: *
Disallow: /shop/
Allow: /shop/sale/

Here /shop/sale/shoes is crawled, because /shop/sale/ is the longer match. /shop/new/ is blocked. The order of the lines does not matter to Google.

Where the File Lives and How Big It Can Be

• The file must be at the root of the host. A robots.txt in a subfolder is ignored.
• Each subdomain, protocol and port needs its own file. blog.example.com does not follow the rules in example.com/robots.txt.
• The file must be plain text in UTF-8.
• Google reads up to 500 KiB. Anything after that is ignored.
• Google usually caches the file for up to 24 hours, so changes are not picked up instantly.

What Happens When the File Returns an Error

How your server responds to the robots.txt request matters a lot:

• 200 (OK): Google reads and follows the file.
• Redirects: Google follows up to five redirects. After that, it treats the file as missing.
• 404 and most other 4xx errors: Google acts as if there is no robots.txt and crawls everything. This is fine if that is what you want.
• 5xx server errors: Google stops crawling the site for the first 12 hours. After that, it uses the last good copy it saved for up to 30 days. If the errors go on longer and the rest of the site works, it treats the file as missing.

That 5xx rule is why I always check that robots.txt loads fast and returns 200. A broken robots.txt can pause crawling of your whole site, even if every page works.

What Google Ignores

• Crawl-delay: Google does not support it. Bing does. Google adjusts its crawl speed by itself based on how your server responds.
• Noindex in robots.txt: Google stopped supporting this in September 2019. It was never an official rule.
• Unknown fields and typos: Google skips lines it does not understand, without warning you in the file itself.

Common robots.txt Mistakes

These are the problems I see most often when I audit sites. Most of them are small typos with big effects.

1. Using robots.txt to Remove Pages From Google

If you block a page in robots.txt, Google cannot crawl it, so it never sees a noindex tag on it. The page can stay indexed, often with the note “No information is available for this page.”

To remove a page, let Google crawl it and add a noindex meta tag (a line in the page’s head that says “do not show this in search”). Once it drops out, you can block it if you still want to. If you need a page hidden fast while that happens, the Search Console Removals tool can hide it for a while. I explain where noindex fits in my on-page SEO checklist.

2. Leaving the Staging Block on After Launch

A Disallow: / copied from a test site to the live site blocks everything. It is one of the most damaging mistakes in SEO and one of the easiest to make during a redesign. I check robots.txt on launch day, every time.

3. Blocking CSS and JavaScript

Google renders pages like a browser. If it cannot load your CSS and JavaScript files, it may not see your content or layout properly. Old robots.txt files often block folders like /wp-includes/ or /assets/. Unless a file is truly useless, let Google load it.

4. Forgetting the Trailing Slash

Disallow: /blog blocks /blog/, but also /blog-tips and /blogroll.html, because rules match the start of the path. Disallow: /blog/ blocks only what is inside the folder. Small slash, big difference.

5. Getting the Case Wrong

If your URLs use /Products/, a rule for /products/ does nothing. Copy paths straight from the address bar instead of typing them from memory.

6. Listing Private Folders

Writing Disallow: /secret-files/ tells everyone where your secret files are. Protect private areas with a login, not with robots.txt.

7. Blocking Pages That Redirect or Carry Canonical Tags

If you block an old URL that redirects to a new one, Google cannot see the redirect. The same goes for canonical tags (a line of code that tells Google which version of a page is the main one). Google has to crawl a page to read these signals, so do not block it.

8. Putting the File in the Wrong Place

A file at example.com/seo/robots.txt does nothing. It must sit at example.com/robots.txt, with that exact lowercase file name.

Robots.txt and AI Crawlers

Many site owners now ask whether they should block AI bots. Robots.txt is the main tool for this, and the big AI companies publish the user-agent names they use. Some common ones:

• GPTBot: OpenAI’s crawler for training data.
• OAI-SearchBot: OpenAI’s crawler for showing sites in ChatGPT search.
• Google-Extended: a control token that tells Google whether your content can help improve Gemini Apps and Vertex AI. Google says it does not affect your inclusion or ranking in Google Search.
• CCBot: Common Crawl, whose data many AI models are trained on.

Blocking a training crawler does not remove content already collected. Blocking a search crawler like OAI-SearchBot can stop your pages from appearing as sources in that tool.

My honest view: decide per bot, based on whether you want the visibility, and write it down in a comment in the file. Blocking everything with one broad rule usually costs more than it protects. AI agents that browse sites like a person are a newer question, which I look at in my piece on OpenAI Operator.

How to Create and Test Your robots.txt File

Here is the process I follow when I set up or fix a robots.txt file.

1. Check what you have now. Open yoursite.com/robots.txt in a browser. Note every rule and ask why it is there.
2. List what should not be crawled. Usually admin areas, cart and checkout, internal search and filter URLs. Leave content pages, CSS and JavaScript open.
3. Write the file in a plain text editor. Save it as robots.txt in UTF-8. Do not use a word processor, because it can add hidden formatting.
4. Add your sitemap line with the full URL.
5. Upload it to the root of each host. On WordPress, an SEO plugin often lets you edit the file without FTP. Check each subdomain separately.
6. Test it. Use the robots.txt report in Google Search Console. It replaced the old robots.txt Tester in November 2023. It shows which robots.txt files Google found, when it last fetched them, and any errors or warnings. You can also ask Google to fetch the file again after a fix.
7. Check key URLs. Use URL Inspection in Search Console on a few important pages to confirm they are not blocked. Then watch the Page indexing report for “Blocked by robots.txt” and “Indexed, though blocked by robots.txt”.

Remember the 24-hour cache. If a fix does not show up right away, give it a day or request a recrawl in the report.

What I Check First in a robots.txt Audit

When I review a site, I do these quick checks before anything else:

• Does /robots.txt return a 200 status and load quickly?
• Is there a Disallow: / under User-agent: * or Googlebot?
• Are any important page types, CSS or JavaScript blocked?
• Do any named groups forget rules from the * group?
• Does the sitemap line point to a working sitemap?
• Are any URLs in the sitemap also blocked? That sends Google mixed signals.

For the full picture of how Google treats blocked pages, Google’s own introduction to robots.txt is short and worth reading.

Final Thoughts on This robots.txt Guide

The key lesson of this robots.txt guide is simple: robots.txt manages crawling, and noindex manages indexing. Keep the file short, block only what wastes crawl time, never block CSS or JavaScript, and test every change in Search Console.

If you are reviewing your whole site, robots.txt is one item on a bigger list. My SEO checklist for 2025 shows what to fix first and where robots.txt fits in.