Google Backs a Shared Robots.txt AI Training Control: What Sites Should Know

If you have been adding a new robots.txt line every time a new AI company launches a crawler, there is finally some good news. On 30 October 2026, Google said it will support a shared AI training control: one robots.txt rule that tells many AI providers at once whether they may use your content to train their models.

The rule is not live in Google’s systems yet. Google says it aims to support it in 2027, and its own Google-Extended control keeps working until then and after. Below, I explain what was announced, how the new rule is written, what it does and does not do, and what I would do with my robots.txt file right now.

What Google Announced on 30 October

Google published a post on the Search Central blog titled “Supporting a shared AI training control for websites”. It was written by John Mueller. The short version is this:

• Google will support a shared “policy token” in robots.txt that lets site owners say whether their content may be used for AI training.
• The idea comes from work in the AI Preferences (AIPREF) working group at the IETF, the body that writes many of the internet’s technical standards.
• Google aims to support the control in 2027, together with other AI providers, and says it will share updates as the work moves on.
• Company-specific controls, including Google-Extended, keep working alongside the shared signal.

Google is not doing this alone. The proposal was written by an Anthropic engineer, and OpenAI and Microsoft have also said they will honour the same signal. That is the big change here. For the first time, the main AI providers are lining up behind one shared opt-out instead of each asking you to block their own bot by name.

Google also tied the move to feedback from publishers, civil society groups and regulators. It said the approach fits the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which treats robots.txt as a main way for publishers to state machine-readable preferences.

Why a Shared AI Training Control Matters

Until now, opting out of AI training has been a game of whack-a-mole. You had to know the name of every relevant bot or token, such as GPTBot, ClaudeBot, Google-Extended and CCBot, and add a separate group for each one. I covered that list in my guide to robots.txt and AI crawlers, and it gets longer every few months.

That approach has three problems I see all the time:

• It mixes up two different questions. Blocking a crawler stops it fetching pages. But many site owners do not mind a bot visiting. They only object to their content being used for training.
• It goes out of date. New bots appear, and old names change. A robots.txt file that was complete last year often is not complete today.
• It is easy to block the wrong thing. I regularly see sites that blocked an AI search bot by mistake and lost visibility in AI answers they actually wanted to appear in.

A shared training control fixes the first two problems. You state your training preference once, in one place, and any provider that supports the standard reads the same rule.

How the New Robots.txt Rule Works

The technical details are in an IETF Internet-Draft called “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal-00). It was published on 30 October 2026 by Kevin Kelley of Anthropic.

The clever part is that it adds no new commands to robots.txt. It reuses the normal format you already know. Instead of a bot name on the User-Agent line, you put a special token that stands for a purpose: AI training.

The Token: __usage_ai-training

The token is __usage_ai-training. It starts with two underscores so it cannot be confused with a real crawler’s name. The draft calls this a “policy token”, meaning it describes how content may be used rather than which bot is visiting.

Under that token, the usual rules apply:

• Disallow on a path means “I prefer this content is not used for AI training”.
• Allow on a path means “AI training is fine for this content”.
• If your file has no group for the token at all, you have expressed no preference.

Example Robots.txt Rules

Here is the simplest opt-out. It asks supporting providers not to train on anything on your site:

User-Agent: __usage_ai-training
Disallow: /

You can also be selective. This example, based on the one in the draft, opts out site-wide but allows training on a press kit folder:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Path matching follows the robots.txt standard (RFC 9309). The most specific, longest matching rule wins, just as it does for Googlebot. So /press-kit/logo.png is allowed, while /blog/ is not.

And this example opts out only a members area, leaving everything else with no stated preference:

User-Agent: __usage_ai-training
Disallow: /members/

What the Wildcard Group Does Not Do

This is the detail I expect many people to miss. The draft says a User-Agent: * group must not be treated as matching the training token. So your existing catch-all rules do not count as a training opt-out.

In practice, that means you must name __usage_ai-training directly. A file that only says User-Agent: * with Disallow: /private/ says nothing about AI training.

Training vs Crawling vs AI Answers

The new token controls one thing: whether your content may be used to train AI models. It is worth being very clear about what sits outside it.

• Crawling. The draft says the token is not a crawling directive. Bots still follow the rules for their own names. A provider may choose not to fetch pages you have opted out of, but if you want to stop a bot from visiting, you still need a group for that bot.
• Google Search. Opting out of training does not remove you from Google Search. Google’s documentation already says Google-Extended does not affect a site’s inclusion in Search and is not a ranking signal.
• AI answers and grounding. Grounding means feeding your live page to an AI model at the time someone asks a question, so it can answer with current facts. This is where providers differ, so read the next section carefully.

Think of it as three switches: can the bot visit, can the content train a model, and can the content be used in live AI answers. The shared token is only the middle switch.

What Happens to Google-Extended

If you followed my earlier guide to Google-Extended, nothing breaks. Google says Google-Extended keeps working alongside the shared signal.

Google-Extended is a robots.txt token, not a separate crawler. Google’s crawler documentation says it controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.

Here is the key rule from Google’s post. If your site uses the shared token to opt out of AI training and does not mention Google-Extended at all, Google will treat it as if you had disallowed Google-Extended.

That has a side effect worth thinking about. Because Google-Extended also covers grounding in Gemini Apps, a plain training opt-out could also keep your pages out of those grounded Gemini answers. Google’s post says sites that explicitly allow Google-Extended can appear in Gemini Apps grounding responses and get clicks from them.

The draft also allows for this. It says a vendor may treat its own named token as more specific than the shared one. So for Google, an explicit Google-Extended rule is the stronger instruction.

Opting Out of Training but Staying in Gemini Answers

If you want to say no to training in general but still want the chance to appear in grounded Gemini answers, you can combine the two. Something like this expresses that intent:

User-Agent: __usage_ai-training
Disallow: /

User-Agent: Google-Extended
Allow: /

Be aware of the trade-off. An explicit Allow for Google-Extended is a Google-specific instruction, and Google-Extended covers both training and grounding for Gemini. You cannot split those two with Google-Extended alone. I would read Google’s documentation again when support goes live in 2027, because the details may be refined before then.

What Other AI Providers Have Said

Google’s blog post only speaks for Google. But the shared control only works if many providers read it, so it matters who else is on board.

• Anthropic: its engineer wrote the draft, and the company has backed the signal.
• OpenAI: has said it plans to support the signal in the coming months.
• Microsoft: has said Bing will honour it, and that the token covers training only. Microsoft handles grounding in Bing’s AI features through separate controls, and says a site’s training choice does not affect its eligibility for Bing results.

Timelines differ by company, and none of this is a law. The draft itself notes that private agreements between a site and a provider may override what robots.txt says. Like all robots.txt rules, it is a request that well-behaved companies choose to respect.

Who Should Use the Shared AI Training Control

My honest view is that this is a business decision, not a technical one. The rule takes two lines. Deciding whether you want it is the hard part.

I would think seriously about opting out if:

• Your content is your product, such as paid research, news, courses, recipes or original data.
• You have licensing deals or plan to sell access to your content.
• Your industry has legal or client rules about how your content may be reused.

I would probably leave training alone, or allow it, if:

• You are a local business, a service company or a store that mainly wants to be found and recommended.
• Your pages are mostly marketing pages that you would happily see repeated by an AI assistant.
• You care more about brand awareness in AI tools than about protecting the text itself.

There is no public evidence that opting out of training hurts how often AI search tools cite you, because citations mostly come from live retrieval, not training. But no provider has promised anything about the long-term effect either. So I would not make big claims in either direction.

What to Do With Your Robots.txt Now

Google will not read the token until 2027, so there is no rush. Still, adding it early does no harm, because parsers that do not understand it simply ignore it like any unknown user agent. Here is the process I would follow.

1. Decide your training preference. Agree with whoever owns your content, legal or business side, whether you want to opt out of AI training site-wide, for certain folders, or not at all.
2. Audit your current file. Open yourdomain.com/robots.txt and list every AI-related group you already have, such as GPTBot, ClaudeBot or Google-Extended. Note which ones block crawling and which only signal training.
3. Add the shared token. Add a User-Agent: __usage_ai-training group with the Disallow and Allow paths that match your decision. Put it in its own group, not mixed into the * group.
4. Decide on Google-Extended separately. If you want to stay eligible for grounded Gemini answers, add an explicit Google-Extended group. If you are happy for Google to treat your opt-out as a Google-Extended block, you can leave it out.
5. Keep your crawler rules where you need them. If you want to stop certain bots from visiting at all, keep those named groups. The shared token does not stop crawling.
6. Test the file. Check the Search Console robots.txt report to make sure Google fetched the file without errors, and check that your Googlebot rules did not change by accident.
7. Review it in 2027. Set a reminder to read each provider’s documentation once support goes live, and adjust if the rules are refined.

If robots.txt syntax is new to you, my robots.txt guide covers groups, paths and the common mistakes I see before you start editing.

Common Mistakes to Avoid

Because the token looks like a bot name, I expect a few predictable errors. These are the ones I would watch for.

• Putting the rule in the wildcard group. Rules under User-Agent: * do not count as a training preference. Use a separate group.
• Thinking it blocks crawling. It does not. Bots may still visit, and server logs will still show them.
• Removing named bot rules too early. Until each provider confirms support, your existing GPTBot or Google-Extended rules are the ones that actually work. I would keep them in place for now.
• Typos in the token. It is two underscores, then usage_ai-training. Matching is exact apart from letter case, so a missing underscore or a space breaks it.
• Blocking Googlebot by accident. Edits to robots.txt are risky. Always compare the old and new file before you publish.
• Confusing it with llms.txt. The llms.txt file is a reading list for AI tools. It cannot opt you out of anything. The training preference lives in robots.txt.

What Is Still Unclear

This is an early proposal, and some questions do not have answers yet.

• Exact launch dates. Google says 2027, with no specific date. Other providers have given their own, different timelines.
• Whether the draft will change. It is the first version (00) of a draft put forward by one author. Drafts often change before the IETF adopts them, and the token name or details could be refined.
• Past training. Nothing in the announcement says an opt-out applies to models that were already trained. I would assume it applies to future use only.
• Other purposes. The token is named for one purpose, AI training. The __usage_ pattern suggests other purposes could get their own tokens later, but nothing has been announced.

Conclusion

The shared AI training control is a small robots.txt change with a big idea behind it. One token, __usage_ai-training, lets you tell Google, OpenAI, Anthropic and Microsoft your training preference in one place. Google plans to support it in 2027, and Google-Extended keeps working, with an opt-out treated as a Google-Extended block unless you say otherwise.

My advice is simple. Decide what you want first, then add the token in its own group and keep your existing bot rules until support is confirmed. As a next step, open your robots.txt file today and compare it against my list of AI crawlers and their tokens, so you know exactly what you are already blocking.