Google-Extended: Blocking Bard and Vertex AI Training From Your Site

If you don’t want your website’s content used to train Google’s AI tools, you now have a simple way to say no. Google-Extended is a new robots.txt token that tells Google not to use your pages to improve Bard and Vertex AI. It is separate from Googlebot, so blocking it does not stop Google crawling and indexing your site for Search.

Google announced the control on 28 September 2023. In this guide I explain what Google-Extended is, what it blocks and what it doesn’t, how to add it to your robots.txt file, and how I would decide whether a site should use it.

What Is Google-Extended?

Google-Extended is a “product token” for robots.txt. A robots.txt file is a plain text file at the root of your site that tells crawlers which parts of the site they may visit. A product token is the name you put on the User-agent line so a rule applies to one specific crawler or product.

Google says publishers can use Google-Extended to manage whether their sites help improve Bard and the Vertex AI generative APIs. That includes future generations of the models that power those products.

In plain words, it is an opt-out switch for AI training and improvement. You add two lines to robots.txt, and Google says it will not use your content for those purposes.

Bard and Vertex AI in One Line Each

• Bard is Google’s AI chatbot. People ask it questions and it writes answers in full sentences.
• Vertex AI is Google Cloud’s platform for businesses. Developers use its generative AI tools to build their own chatbots and apps.

If you want the background on Bard itself, I covered it in what Google’s Bard announcement means for search and later in my first look at Bard’s early access.

Why Google Launched It Now

Publishers have been asking a fair question all year: if AI companies learn from my content, do I get any say in it? Until now, the only clear answer was to block a crawler completely, which also meant leaving search.

In July 2023, Google opened a public discussion about new ways for web publishers to control how their content is used for AI. Google-Extended is the first concrete result of that.

Other companies moved in the same direction. In August 2023, OpenAI published details of GPTBot, its web crawler, and explained how to block it in robots.txt. Common Crawl, a public web archive that many AI models have learned from, has long had its own crawler called CCBot. And Bing announced options for controlling how content is used in Bing Chat about a week before Google’s announcement.

Google’s post also says it will keep working with the web and AI communities on more machine-readable ways to give publishers choice. So I treat Google-Extended as a first step, not the final answer.

What Google-Extended Blocks and What It Doesn’t

This is the part most people get wrong, so it is worth going slowly.

What It Blocks

• Use of your content to help improve Bard.
• Use of your content to help improve the Vertex AI generative APIs.
• Use of your content for future generations of the models that power those products.

What It Doesn’t Block

• Google Search. Google points site owners to Googlebot rules and noindex for managing Search. Google-Extended is a separate control for AI use.
• Crawling by Googlebot. Googlebot keeps visiting your pages as normal.
• Search Generative Experience. Google has said that site owners should keep using Googlebot rules and the noindex tag to manage Search, and that this includes the Search Generative Experience (SGE) tests in Search Labs.
• Other AI companies. GPTBot, CCBot and other crawlers each need their own rules.

That third point surprises many site owners. If your main worry is your content showing up in Google’s AI answers inside search results, Google-Extended is not the tool for that today. If you want to know how those AI answers work, see my explainer on the Search Generative Experience from Google I/O 2023.

The only way to keep a page out of SGE right now is to keep it out of Search, using noindex or a Googlebot block. For almost every business, that cost is far too high.

Google-Extended Is a Token, Not a New Crawler

Some early reports called Google-Extended a “new Googlebot”. It isn’t.

Google’s crawler documentation says Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with Google’s existing user agents, and the Google-Extended token in robots.txt is used in a “control capacity”. In other words, it tells Google what it may do with content it has already fetched.

This has two practical results:

• You will not see “Google-Extended” in your server logs. Don’t go looking for it.
• Blocking Google-Extended does not save crawl budget or server load. Googlebot still fetches the same pages.

This works a bit like the Googlebot-News token, which controls how content is used in Google News rather than how it is crawled.

How to Block Google-Extended in robots.txt

The setup takes a few minutes. Here is the process I follow.

1. Open your current robots.txt. Visit yourdomain.com/robots.txt in a browser and copy what is there. Every host (such as www and shop subdomains) has its own file.
2. Decide on full or partial blocking. Choose whether to block the whole site or only some folders.
3. Add a Google-Extended group. Add a new block that starts with User-agent: Google-Extended, followed by your Disallow rules.
4. Leave your Googlebot rules alone. Don’t edit the User-agent: Googlebot or User-agent: * groups unless you mean to change Search.
5. Upload the file. Save it to the root of the site, or update it through your CMS or SEO plugin.
6. Check the live file. Load yourdomain.com/robots.txt again and confirm the new lines are there and nothing else changed.

Example: Block the Whole Site

User-agent: Google-Extended
Disallow: /

This tells Google not to use any page on this host to improve Bard or Vertex AI. Search is not affected.

Example: Block Only Some Sections

User-agent: Google-Extended
Disallow: /guides/
Disallow: /research/

This suits a site that is happy for general pages to be used but wants to protect its most valuable work, such as paid research or in-depth guides.

Example: A Full File With Other AI Crawlers

User-agent: *
Disallow: /cart/
Disallow: /account/

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Googlebot has no group of its own here, so it follows the * rules and crawls the site for Search as usual.

A robots.txt Mistake I’d Watch For

Under the robots.txt standard, a crawler follows only the most specific group that matches its name. Once you add a User-agent: Google-Extended group, the rules under User-agent: * no longer apply to that token.

So if your * group blocks /private/ and your Google-Extended group only blocks /guides/, then /private/ is not covered for Google-Extended. If you want both, list both folders in the Google-Extended group.

The bigger risk is a slip that hits Search. When I review robots.txt changes, the first thing I check is that nobody typed Disallow: / under Googlebot or * by accident. One wrong line there can drop a site out of Google.

A few other habits I’d recommend:

• Keep a dated copy of the old file before you edit it.
• Write the token exactly as Google-Extended. A typo means the rule is ignored.
• Remember robots.txt rules match from the start of the path, so /guides/ covers everything inside that folder.
• Watch the Crawl stats and Page indexing reports in Search Console for a week or two after any robots.txt change.

Should You Block Google-Extended?

There is no single right answer. It depends on how you make money from your content and what you think AI tools will do for your brand.

Reasons to Block It

• Your content is the product. News sites, research firms, course creators and paid newsletters may not want their work used to train tools that answer the same questions.
• You have licensing plans. If you might license your content to AI companies one day, giving it away now weakens that position.
• The Search risk looks low. Googlebot still crawls your pages as normal, and Google treats Search as a separate control, so I don’t expect blocking to hurt rankings.

Reasons to Allow It

• You want your brand known to AI tools. If people ask Bard about your industry, you may want the model to know your products, terms and point of view.
• Your content is mostly marketing. Product pages, service pages and help docs exist to spread information. Many businesses are happy for that to travel widely.
• You don’t sell content. For a local plumber or a small online shop, there is little to protect and little to gain from blocking.

Questions to Ask Before You Decide

When I help a team make this call, I ask a short set of questions. The answers usually point clearly one way.

1. Do we earn money when people read our content on our own site? If yes, through ads, subscriptions or paid access, blocking deserves a serious look.
2. Is our content unique? Original research, data and expert analysis are worth more than general explainers that exist on hundreds of other sites.
3. Do we want AI tools to describe our brand and products correctly? If yes, allowing access may help over time.
4. Are we likely to license our content? If that conversation is possible, keep your options open by blocking for now.
5. Who owns this decision? Write down who agreed to it and why, so the next person who edits robots.txt understands the rule.

My Take

In my experience, most small and medium business sites don’t need to block Google-Extended. Their pages are built to inform customers, and being understood by AI tools is more likely to help than hurt.

For publishers whose income depends on people reading the content on their own site, I’d lean towards blocking, at least for the most valuable sections. The partial block is a sensible middle path, and you can change your mind later by editing two lines.

Whatever you decide, make it a deliberate business choice, not a reaction to headlines. I’d bring the content owner and whoever handles licensing into the decision, not just the SEO team.

What Google-Extended Does Not Settle Yet

Google-Extended is useful, but it leaves some questions open. I would keep these in mind before telling anyone it “protects” their content.

• Past training. Google’s announcement is about what happens from now on. It does not say how content used before the token existed will be treated, so I wouldn’t assume it is removed from existing models.
• AI answers in Search. SGE is managed with Search controls, not Google-Extended. A separate opt-out for AI answers in Search does not exist today.
• Other companies. Each AI company uses its own crawler name. There is no single line that blocks every AI tool.
• Copies of your content. If other sites scrape and republish your pages, your robots.txt does not cover those copies.

Google says it plans to share more about further publisher controls, so I expect this area to keep changing.

How Google-Extended Fits With Your Wider SEO Plan

For most sites, the main job hasn’t changed: publish content that is genuinely useful to people. Google’s guidance on that was updated only this month, and I went through the changes in my piece on the September 2023 helpful content update.

Google-Extended sits next to that work as a content-rights decision. It is closer to a licensing choice than an SEO tactic.

Here is how I would sort the controls you have today:

• To manage Google Search and SGE: use Googlebot rules in robots.txt and the noindex tag.
• To manage use for Bard and Vertex AI: use the Google-Extended token.
• To manage OpenAI’s crawler: use the GPTBot token.
• To manage Common Crawl: use the CCBot token.

Keeping these separate in your head avoids the most common mistake I expect to see this autumn: someone blocks Googlebot to stop AI training and loses their search traffic.

Conclusion

Google-Extended gives publishers a clear, low-risk way to stop Google using their content to improve Bard and Vertex AI. It is two lines in robots.txt, it leaves Googlebot crawling as normal, and it does not cover SGE in search results.

Your next step is simple: open your robots.txt, decide whether your content is something you sell or something you want spread, and add the Google-Extended group only if blocking fits that answer. You can read Google’s own announcement in An update on web publisher controls.