Robots Meta Tags and X-Robots-Tag: noindex, nofollow and More Explained

Google is showing a page you want hidden, or ignoring a page you want found. In that case, the robots meta tag is usually the first place I look. A robots meta tag is a single line of HTML. It tells search engines whether to index a page, follow its links and show a snippet. Its twin, the X-Robots-Tag HTTP header, does the same job for files that have no HTML, such as PDFs and images.

In this guide I cover every rule Google supports, how to use them on HTML pages and on files, and the mistakes that quietly stop them from working.

What a Robots Meta Tag Is and Where It Goes

A robots meta tag sits in the <head> of a page. It has two parts. A name says which crawler it’s for. A content value holds one or more rules.

<!DOCTYPE html>
<html>
<head>
  <meta name="robots" content="noindex">
  <title>Thank you for your order</title>
</head>

Using name="robots" speaks to every search engine crawler. Both the attribute names and the values ignore upper and lower case, so NOINDEX and noindex mean the same thing.

The tag is a page-level instruction. It controls how a URL is indexed and shown in search results. It doesn’t control whether a crawler can fetch the URL. That’s the job of robots.txt. In my experience, mixing up the two causes more indexing problems than anything else on this list.

Every Robots Meta Tag Rule Google Supports

Google lists a fixed set of indexing and serving rules. Google Search ignores anything outside this list. You can combine rules with commas in one tag, or split them across several tags.

Noindex

noindex keeps the page out of search results. Google can still crawl it, but it won’t show it. Typical examples are thank-you pages, internal search results, thin tag archives, staging copies and account pages.

Use it when you want a page to exist for visitors but not appear in Google. If you want it gone completely, removing the page or putting it behind a password is the stronger fix.

Nofollow

nofollow tells Google not to follow the links on the page. Without it, Google may use those links to find other URLs.

This is different from the rel="nofollow" attribute, which applies to a single link. The meta tag version covers every link on the page at once. It’s rarely needed on normal content pages. Cutting off all internal links also cuts off Google’s path to your own content.

None

none is short for noindex, nofollow. It saves a few characters. But I prefer writing both rules out, because anyone reading the code later understands it straight away.

All

all means no limits. It’s the default, so adding it changes nothing. You don’t need index, follow or all on pages you want indexed. Leaving the tag out has the same effect.

nosnippet

nosnippet stops Google from showing a text snippet or video preview for the page. A still image thumbnail may still appear if Google thinks it gives a better experience.

Google’s documentation says this rule applies across web search, Google Images and Discover, and to AI Overviews and AI Mode. More on that below.

Max-Snippet:[number]

max-snippet limits the text snippet to a set number of characters. For example, max-snippet:160 allows up to 160 characters.

• max-snippet:0 works like nosnippet.
• max-snippet:-1 lets Google choose the length.
• If the value isn’t a number Google can read, the rule is ignored.

It limits text only. It doesn’t affect image or video previews.

Max-Image-Preview:[setting]

This sets the largest image preview Google may show. The values are none, standard and large. large can be as wide as the screen.

For most publishers, max-image-preview:large is the setting you want. Large preview images matter for visual places like Discover. Setting none or standard by accident can make your results look smaller than competitors’. This is one of the first things I check on news and blog sites.

Max-Video-Preview:[number]

This limits video previews in seconds. 0 allows at most a still image, and -1 means no limit.

notranslate

notranslate stops Google from offering a translated version of the page in search results. Use it if machine translation would twist your content, such as legal text or brand copy.

noimageindex

noimageindex stops the images on the page from being indexed. Be careful: if the same image is also used on another page without this rule, it may still be indexed through that page.

unavailable_after:[date/time]

This removes the page from search results after a set date and time. It suits limited-time offers, event pages and job listings that expire.

<meta name="robots" content="unavailable_after: 2026-03-31T23:59:00+00:00">

Google accepts common formats such as RFC 822, RFC 850 and ISO 8601. Invalid dates are ignored. After the date passes, Googlebot also crawls the URL far less often.

indexifembedded

indexifembedded only works together with noindex. It lets Google index the content when it’s embedded in another page through an iframe (a frame that shows one page inside another) or a similar tag. The standalone URL is still kept out of results.

This is useful for media players, widgets and embeddable maps. The standalone URL stays hidden, but the content counts as part of the pages that embed it.

<meta name="googlebot" content="noindex, indexifembedded">

Rules Google No Longer Uses

You’ll still find noarchive in plenty of templates. Google Search no longer uses it, because the cached page link it controlled has gone. nocache is also ignored. They do no harm, but they do nothing in Google either. Other search engines may still read them.

Combining Rules and Handling Conflicts

You can write several rules in one tag:

<meta name="robots" content="max-snippet:50, max-image-preview:large">

Or spread them across separate tags:

<meta name="robots" content="nofollow">
<meta name="robots" content="max-snippet:50">

When rules conflict, Google applies the stricter one. If a page has both nosnippet and max-snippet:50, Google treats it as nosnippet.

This catches people out when a theme and an SEO plugin both print a robots tag. One says index, the other says noindex, and noindex wins. I always check the final source code, not just the plugin settings.

Bot-Specific Robots Meta Tags

You can replace robots in the name attribute with a specific crawler’s user agent token (the name a bot uses). Google recognises two tokens for this in Search:

• googlebot for all text results
• googlebot-news for news results

For example, this keeps full snippets in web results but removes snippets from news results:

<meta name="googlebot-news" content="nosnippet">

Crawlers that aren’t part of Search, such as AdsBot-Google, must be named directly if you want a rule to apply to them. Other search engines read their own tokens. So Google ignores a tag for another engine’s crawler, and the other way round.

If a general tag and a bot-specific tag disagree, Google combines the strict rules. A page with robots: nofollow and googlebot: noindex is treated by Googlebot as noindex, nofollow.

The X-Robots-Tag HTTP Header

A PDF, an image or a video file has no <head>, so you can’t add a meta tag to it. The X-Robots-Tag header solves that. Your server sends it with the file. Any rule that works in a robots meta tag also works here.

A response that keeps a file out of search looks like this:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

You can send several rules as a comma-separated list or as separate headers. You can also target a specific crawler by putting its token first:

X-Robots-Tag: noimageindex
X-Robots-Tag: unavailable_after: 2026-06-30T12:00:00+00:00
X-Robots-Tag: googlebot: nofollow

Apache Example: Noindex Every PDF

Add this to your .htaccess file or server configuration. It needs the mod_headers module.

<Files ~ ".pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

Apache Example: Noindex Image Files

<Files ~ ".(png|jpe?g|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>

NGINX Example: Noindex Every PDF

location ~* .pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

NGINX Example: One Specific File

location = /downloads/price-list-2025.pdf {
  add_header X-Robots-Tag "noindex";
}

The header is also handy for HTML pages when editing templates is hard. You can apply a rule to a whole folder, such as a staging folder, from the server config in one go.

data-nosnippet: Hide Part of a Page From Snippets

Sometimes you’re happy for a page to rank but don’t want one section quoted. The data-nosnippet attribute marks text that Google shouldn’t use in a snippet.

<p>Our standard delivery time is 3 to 5 working days.</p>
<div data-nosnippet>
  <p>Trade prices are shown only to approved accounts.</p>
</div>

A few rules make it work reliably:

• It works on span, div and section elements only. Wrap custom elements in one of these.
• It’s a boolean (on/off) attribute. Any value you give it, including "false", is ignored. Just having it there switches it on.
• Your HTML must be valid. An unclosed div can spread the rule over everything that follows.
• Don’t add or remove it with JavaScript on existing elements. If you insert new elements with JavaScript, include the attribute when they are created.

Structured data inside a data-nosnippet block can still be used for rich results. If you need to stop that, edit or remove the structured data itself.

How Snippet Controls Apply to AI Overviews and AI Mode

Many publishers want to know if they can keep content out of Google’s AI answers without leaving Search. Google’s documentation says the existing preview controls apply to its AI features in Search. There’s no separate “no AI” meta tag for them.

• nosnippet applies to AI Overviews and AI Mode, as well as regular results.
• max-snippet limits how much of your text can be shown. Google’s docs say it also applies in these AI features.
• data-nosnippet keeps marked sections out of previews.
• noindex removes the page from Search completely, including these features.

Google’s AI features and your website page lists these same four controls for limiting what’s shown from your pages. It also makes clear that Google-Extended isn’t the tool for this. Google-Extended controls whether your content is used for training and grounding in Google’s other AI products. It doesn’t control whether it appears in Search features.

Think about the trade-off before you act. Limiting snippets reduces what Google can show from your page everywhere, not only in AI answers. So a blanket nosnippet can make your regular listings much less appealing. In my opinion, a targeted data-nosnippet on the sensitive section is usually the better choice.

Gemini 3 now powers AI Overviews, as covered in my guide to Gemini 3 in AI Overviews. But the controls themselves haven’t changed.

Robots.txt vs Noindex: Why Blocked Pages Keep Showing Up

Here’s the trap. Google only sees a robots meta tag or X-Robots-Tag header when it crawls the URL. If robots.txt blocks crawling, Google never fetches the page. It never sees the noindex, so it can’t obey it.

A blocked URL can still be indexed if other pages link to it. It then appears in results with little or no description. That’s exactly what you were trying to avoid.

• Use robots.txt to manage crawling, such as stopping bots from wasting time on endless filter combinations.
• Use noindex to keep a page out of results. The page must stay crawlable so Google can read the rule.
• Don’t combine them on the same URL if your goal is removal. Remove the disallow, let Google crawl again and see the noindex, then decide whether to block it later.

Also note that Google stopped supporting noindex lines inside robots.txt files in 2019. If your robots.txt still has one, it does nothing in Google.

A Word on JavaScript

Google’s JavaScript SEO advice warns that when Googlebot finds noindex in the original HTML, it may skip rendering. So a script that removes the tag later may never run. If you want a page indexed, don’t ship noindex in the first HTML and rely on JavaScript to take it out.

Robots Meta Tag Settings in WordPress

WordPress handles most of this for you. But there are three places I always check.

1. Settings > Reading. The “Discourage search engines from indexing this site” box adds a sitewide noindex robots tag. It’s meant for sites still being built. Forgetting to untick it after launch is one of the most common reasons a new site never ranks.
2. Your SEO plugin’s per-page settings. Most SEO plugins add an advanced panel to each post and page. There you can set noindex, nofollow and snippet rules for that URL.
3. Your SEO plugin’s archive settings. These control whole groups of URLs, such as tag archives, author archives, date archives and media attachment pages. Decide which of these deserve to be in search, and noindex the rest.

Recent WordPress versions also print max-image-preview:large by default on public sites. Leave that in place unless you have a reason to limit image previews.

For PDFs in your media library, WordPress has no built-in noindex switch. Use the X-Robots-Tag examples above in your server config, or ask your host to add them.

How to Test Your Robots Meta Tags

Never assume a rule is live because a settings screen says so. Check what Google actually receives.

1. View the source. Open the page, view source and search for name="robots" and name="googlebot". Look for duplicate or conflicting tags.
2. Check the headers. For PDFs and images, look at the HTTP response in your browser’s developer tools (Network tab) and find X-Robots-Tag.
3. Run URL Inspection in Search Console. Enter the URL and look at “Indexing allowed?”. It tells you whether a noindex was found in the meta tag or the header.
4. Use the live test. Click “Test live URL” to see the current version. Then click “View tested page” to check the rendered HTML and the response headers Google received.
5. Request indexing once you’ve fixed a problem, so Google crawls again sooner.
6. Watch the Page indexing report. The “Excluded by ‘noindex’ tag” list shows every URL Google found with the rule. Scan it for pages that should be indexed. I make this a regular habit.

Removals aren’t instant. Google has to crawl the URL again before a new rule takes effect. That can take anywhere from days to months, depending on how often the page is crawled. My walkthrough of Search Console’s weekly and monthly views helps you track whether those changes show up in your traffic.

Common Robots Meta Tag Mistakes

• Staging settings copied to live. A sitewide noindex on a staging site travels with the database or template to the live site. I see this often after a site launch or redesign.
• Noindex plus a robots.txt block. Google can’t see the rule, so the URL stays in results.
• Noindex on paginated or category pages that carry your only links to deeper content. Over time, this slows down how Google finds those pages.
• Canonical and noindex pointing different ways. A page that says “my canonical is page B” and also “do not index me” sends mixed signals. Pick one approach.
• Accidental max-image-preview:none or a very low max-snippet value set in a theme, which shrinks your listings.
• Expecting nofollow to hide a page. It only affects links on that page. Other pages linking to the URL can still lead Google to it.

Conclusion

The robots meta tag gives you precise control over what Google indexes and shows. The X-Robots-Tag header extends that control to PDFs, images and whole folders. Keep pages crawlable when you use noindex. Prefer targeted snippet controls over blanket ones. And confirm every change in URL Inspection instead of trusting a settings screen.

For the full list of rules, keep Google’s robots meta tag specification bookmarked. Then make an indexing audit part of your 2026 SEO plan, so stray noindex tags never sit unnoticed for months.