Yes, you can block AI crawlers with robots.txt. But it only works for bots that publish a name and promise to follow the file. The real question is which ones to block. Training bots and AI search bots now have different names. So you can say no to model training and still show up in ChatGPT, Claude or Perplexity answers.
In this guide I explain how robots.txt works and list the AI user agents exactly as their owners name them. I also give you rules you can copy, and show you how I check server logs to see if the rules work.
How robots.txt Works (the Parts That Matter for AI Bots)
Robots.txt is a plain text file at the root of your site, such as https://example.com/robots.txt. Crawlers that follow the standard read it before they fetch other URLs. They skip the paths you block. It is not the same file as llms.txt, which I cover in my LLMs.txt guide.
Robots.txt controls crawling. It does not lock anything. Google’s own documentation says it is not a way to keep a page out of Google, and that not every crawler obeys it. Keep that in mind for everything below.
Groups, User Agents and Rules
A robots.txt file is made of groups. Each group starts with one or more User-agent lines. Each line names a crawler. Then come Disallow and Allow rules for that crawler. For the full syntax and common mistakes, see my robots.txt guide.
- Disallow: / blocks the whole site for that crawler.
- Disallow: /members/ blocks one folder and everything inside it.
- Allow makes an exception inside a blocked path.
- An empty Disallow: line, or no group at all, means the crawler can fetch everything.
The Rule Most People Get Wrong
A crawler follows only one group. It picks the group with the most specific user agent that matches it. Say you have a User-agent: * group and a separate User-agent: GPTBot group. GPTBot ignores the asterisk group completely.
So if your general group blocks /cart/ and /search/, and you add a GPTBot group that only allows a few paths, GPTBot can now crawl your cart and search URLs. Copy any rules you still want into each named group. This is the first thing I check when I review a robots.txt file with AI rules in it.
Scope and Timing
- Each host needs its own file. Rules on
www.example.comdo not covershop.example.comorblog.example.com. Anthropic’s help page says this clearly for its own bots. - Changes are not instant. OpenAI and Perplexity both say their systems can take about 24 hours to pick up a robots.txt change. Google also usually caches the file for up to a day.
- Blocking only works from now on. It tells a company not to collect or use new crawls. It does not pull your content out of models or datasets built before.
If you want a quick refresher on how crawling, indexing and ranking fit together, see my explainer on how Google Search works in 2026.
Training Crawlers vs Search and Retrieval Crawlers
Most AI companies now split their bots by job. Once you know the three jobs, every robots.txt choice gets easier.
- Training crawlers collect public pages that may be used to train future models. Blocking them costs you nothing in that company’s AI search product, as long as the search bot is a separate bot.
- Search or indexing crawlers build the index an AI assistant searches when it answers a question and cites sources. If you block these, your pages usually stop showing up as cited links in that assistant.
- User-triggered fetchers visit a page because a person asked the assistant to read it or look something up. Several companies say these may not follow robots.txt, because a human made the request.
The trade-off is simple. Blocking training bots keeps your content out of models, but you get no traffic back. Blocking search bots removes a growing source of visits and a chance to be the brand an assistant cites.
For most businesses that need to be found, I think the sensible default is this: decide on training based on your content policy, and keep search crawlers allowed. Publishers whose content is the product may fairly choose to block more.
The Main AI User Agents, as Documented by Their Owners
Every name below comes from the owner’s own documentation, checked in April 2026. Use these exact names. Robots.txt does not care about upper or lower case in the user agent, but spelling matters.
| Token | Owner | Role | Follows robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Training | Yes |
| OAI-SearchBot | OpenAI | ChatGPT search | Yes |
| ChatGPT-User | OpenAI | User-triggered | May not apply |
| Google-Extended | Gemini training and grounding (control token only) | Yes | |
| ClaudeBot | Anthropic | Training | Yes |
| Claude-SearchBot | Anthropic | Search quality | Yes |
| Claude-User | Anthropic | User-triggered | Yes |
| PerplexityBot | Perplexity | Search indexing | Yes |
| Perplexity-User | Perplexity | User-triggered | Generally ignores |
| Applebot-Extended | Apple | AI training (control token only) | Yes |
| CCBot | Common Crawl | Public web archive | Yes |
| Meta-ExternalAgent | Meta | Training and product indexing | Yes |
OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User
OpenAI lists its three main bots on its crawler overview page. You control each one separately.
- GPTBot crawls content that may be used to train OpenAI’s foundation models. Blocking it tells OpenAI your content should not be used for training.
- OAI-SearchBot shows websites in ChatGPT’s search features. OpenAI says sites that block it will not be shown in ChatGPT search answers, though they may still appear as plain navigation links.
- ChatGPT-User handles actions users start in ChatGPT and custom GPTs, such as visiting a page to answer a question. OpenAI says robots.txt rules may not apply to it, and that it is not used to decide what appears in search.
If you allow both GPTBot and OAI-SearchBot, OpenAI says it may use one crawl for both jobs, so it does not fetch your pages twice. Your GPTBot training rule also covers pages people view in OpenAI’s ChatGPT Atlas browser, which has no user agent of its own.
Google-Extended: What It Does and Does Not Control
Google-Extended is the most misunderstood name on this list. It is not a crawler. Google says it has no separate HTTP user agent string. Google’s normal crawlers do the fetching, and the token only works as a switch in robots.txt. I covered the original announcement in my post on Google-Extended and Bard training.
Google’s common crawlers documentation says it lets you choose whether content Google crawls may be used to train future Gemini models and for grounding. Grounding means content from Google’s index is given to Gemini when it answers a prompt.
What it does not do matters just as much:
- It does not change whether your site is in Google Search. Google says it is not a ranking signal.
- It does not keep your pages out of AI Overviews or AI Mode. Those are Google Search features, crawled by Googlebot.
- You will never see it in your server logs, because nothing calls itself Google-Extended.
For AI features in Search, Google’s controls are the normal Search ones: nosnippet, data-nosnippet, max-snippet and noindex. These also limit what shows in regular results, so use them with care. My comparison of AI Overviews and AI Mode explains how the two features use your content.
Never block Googlebot to stay out of AI Overviews. You would be taking yourself out of Google Search. I still see people suggest this online, and it is the one mistake here I would call truly costly.
Anthropic: ClaudeBot, Claude-SearchBot and Claude-User
Anthropic’s help centre lists three bots and says all of them follow robots.txt.
- ClaudeBot collects public web content that may help train models. Blocking it tells Anthropic to leave your future content out of training data.
- Claude-SearchBot crawls to improve search results for Claude users. Blocking it means your content is not indexed for search, which can lower your visibility in Claude’s answers.
- Claude-User fetches pages when a Claude user asks a question that needs them. Blocking it stops Claude from getting your pages for a user.
Anthropic also supports the non-standard Crawl-delay rule. This is handy if you want to slow ClaudeBot down instead of blocking it. Older lists online still include names like anthropic-ai and Claude-Web. These are not among the bots on Anthropic’s current page.
Perplexity: PerplexityBot and Perplexity-User
PerplexityBot finds and links websites in Perplexity’s search results. Perplexity says it is not used to crawl content for AI foundation models. So blocking it mostly costs you citations.
Perplexity-User visits pages when a user asks a question. Perplexity says this fetcher generally ignores robots.txt because the request comes from a person. If you need to stop it, robots.txt is not the right tool.
Applebot-Extended
Apple works the same way as Google. Applebot is the real crawler, and its data powers search in Spotlight, Siri and Safari. Applebot-Extended does not crawl. It only tells Apple whether data Applebot already collected can be used to train its generative AI models.
Apple says blocking Applebot-Extended does not remove pages from its search results. Blocking Applebot itself does.
CCBot and Meta-ExternalAgent
CCBot is the crawler of Common Crawl, a non-profit that gives a free copy of the web to researchers and companies. Its archive has been widely used to build AI training datasets. That is why it is on most blocklists. It follows robots.txt and Crawl-delay.
Meta-ExternalAgent crawls for uses such as training foundation AI models and improving Meta products by indexing content. Meta also has Meta-ExternalFetcher, which fetches links when a user asks. Meta itself says it may bypass robots.txt.
Example robots.txt Rules to Block AI Crawlers
Each example below is a set of lines to paste into your robots.txt file. Put one line per item, in the order shown. Add them next to your existing groups. Do not replace the whole file.
Example 1: Block AI Training, Keep AI Search
This is the setup I suggest most businesses look at first. It blocks the training bots and control tokens but leaves search crawlers and Googlebot alone.
User-agent: GPTBotUser-agent: ClaudeBotUser-agent: Google-ExtendedUser-agent: Applebot-ExtendedUser-agent: CCBotUser-agent: Meta-ExternalAgentDisallow: /
Putting several User-agent lines above one set of rules is valid under the robots.txt standard. If you want to be safe with simpler parsers, give each bot its own two-line group. That is what I usually do, because it is also easier to read later.
Example 2: Block AI Search and Retrieval Too
Add this group if you do not want AI assistants to cite or fetch your pages. Expect fewer visits from those tools.
User-agent: OAI-SearchBotUser-agent: ChatGPT-UserUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: PerplexityBotUser-agent: Perplexity-UserDisallow: /
Remember, ChatGPT-User and Perplexity-User may not obey this. Listing them states your wish, but you need server rules to enforce it.
Example 3: Protect Only Certain Sections
You do not have to choose all or nothing. This keeps training bots out of premium and members areas but leaves public marketing pages open.
User-agent: GPTBotUser-agent: ClaudeBotUser-agent: CCBotDisallow: /members/Disallow: /reports/Disallow: /cart/
Look at the repeated /cart/ rule. These bots now match this group, so they no longer read your User-agent: * rules.
Example 4: Slow a Crawler Down Instead of Blocking It
If the problem is server load, not principle, a crawl delay may be enough for bots that support it, such as ClaudeBot and CCBot.
User-agent: ClaudeBotCrawl-delay: 2
Googlebot and Applebot ignore Crawl-delay, so this does nothing for them.
The Limits of robots.txt
Robots.txt is a public request, not a lock. It works because trusted companies choose to follow it. It has several gaps you should plan for.
- Following it is optional. Unnamed or badly behaved scrapers can ignore the file. Some may even use it as a map of what you want hidden.
- User-triggered fetchers may not follow it. OpenAI, Perplexity and Meta all say their user-started fetchers may not apply robots.txt rules.
- User agents can be faked. Anyone can send a request labelled GPTBot. Rules based only on the user agent block honest bots and fakes alike. They also miss scrapers that pretend to be a browser.
- It does not remove past data. Content collected before you added the rule may already be in a dataset.
- New bots appear. A list written today will miss names launched next quarter. I review mine a few times a year, and I suggest you do the same.
Server-Level Blocking
When you need real enforcement, block at the server, CDN or firewall. This sends an error back to the request instead of asking the bot to stay away. Cloudflare, for example, now blocks AI crawlers by default on new domains, which I covered in Cloudflare’s AI crawler blocking.
On Nginx, a simple user agent rule inside your server block looks like this:
if ($http_user_agent ~* "(GPTBot|ClaudeBot|CCBot|Perplexity-User)") { return 403; }
Apache users can do the same with a rewrite condition on HTTP_USER_AGENT. Most CDNs and web application firewalls also have bot rules you can switch on without touching server code.
For stronger control, match the user agent with the IP ranges the companies publish. OpenAI, Anthropic, Perplexity, Apple and Common Crawl all publish JSON lists of their crawler IP addresses, linked from their bot documentation. Apple and Common Crawl also support reverse DNS checks (looking up the name behind an IP address).
Be careful with broad firewall rules. Blocking a whole cloud provider’s IP range, or every unknown bot, can catch Googlebot, Bingbot, uptime monitors and payment webhooks. Test on a staging site or in log-only mode first. In my experience, this is where most self-inflicted damage happens, not in robots.txt itself.
How to Check Your Logs for AI Crawlers
Your server access logs show which bots really visit and whether they follow your rules. Analytics tools such as GA4 usually do not record bots, so logs are the only reliable view. My beginner’s guide to log file analysis goes further.
- Download recent access logs from your host, server or CDN. A week or two is enough to start.
- Search for the AI names. On Linux or macOS this command counts hits per bot:
grep -ioE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|ccbot|applebot|meta-externalagent" access.log | sort | uniq -c | sort -rn - Check which URLs each bot asks for and whether it fetched
/robots.txtfirst. - Compare requests made after your change against the paths you blocked. Give caches a day to update first.
- For any bot that seems to break the rules, check its IP against the owner’s published IP list. If it does not match, it is a fake using the bot’s name.
Do not search your logs for Google-Extended or Applebot-Extended. They never appear, because Googlebot and Applebot do the crawling.
Logs show crawling, not visits from people. To see the visits AI assistants send you, read my guide to tracking ChatGPT, Perplexity and Gemini visits in GA4.
Should You Block AI Crawlers at All?
There is no single right answer. But a few questions make the choice clearer.
- Is your content the product? Subscription publishers, research firms and course creators lose more from free reuse. Blocking training bots, and maybe search bots, makes more sense for them.
- Do you sell services or products? Being cited by an assistant puts your brand in front of buyers. Blocking search crawlers usually hurts more than it helps.
- Is server load the real problem? Try crawl delays and rate limits before a full block.
- Do you have legal or licensing duties? Content you licence from others may need to be blocked, whatever your own view is.
If you decide to stay visible, the next job is making your pages easy to quote. My guide on writing content AI assistants can quote covers the answer-first format.
Conclusion
To block AI crawlers with robots.txt well, sort the bots by job. If you do not want your content used for training, block training crawlers like GPTBot, ClaudeBot and CCBot, plus the Google-Extended and Applebot-Extended tokens. Keep search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed if you want citations.
Then treat robots.txt as a stated wish. Back it with server rules where it matters, and check your logs after every change. My advice: start today by opening your live robots.txt file. Make sure none of your new AI groups has quietly dropped the rules from your User-agent: * group.
