Are your pages missing from Google? Or do they show up, but never where you want them? In my experience, the cause almost always sits in one of three stages. Here is how Google Search works in plain words. Googlebot crawls the pages it finds. Google indexes the ones worth keeping. Then its ranking systems serve the most useful results for each search. AI Overviews and AI Mode are built on top of that same pipeline, not beside it.
How Google Search Works: The Three Stages
Google describes Search as three stages, and it is clear that not every page gets through all of them. It also says plainly that it does not take payment to crawl a site more often or to rank it higher.
| Stage | What Google does | Common reason a page drops out |
|---|---|---|
| Crawling | Finds URLs and downloads their text, images, video and code | Blocked by robots.txt, server errors, no links pointing to it |
| Indexing | Renders the page, works out what it is about, picks a canonical version and stores it | Low quality, copy of another URL, noindex tag |
| Serving | Matches indexed pages to a search and puts them in order | Not relevant enough, weaker than competing pages |
Google’s own in-depth guide to how Google Search works is the main source for this model. When a page is not performing, my first step is always to find which of these three stages is stopping it.
Stage 1: Crawling and How Googlebot Finds Your Pages
There is no master list of every page on the web. Google has to keep finding URLs, and it does that with an automated program called Googlebot.
How Discovery Works
Googlebot finds new URLs in three main ways:
- Links from pages it already knows. A new blog post linked from your homepage or a category page gets found the next time Google crawls that page.
- Links from other sites. A mention on a supplier’s site or an industry directory can lead Googlebot to you.
- XML sitemaps. A file that lists the URLs you want crawled. You submit it in Search Console or point to it in robots.txt.
How Google Decides What to Crawl and How Often
An algorithm decides which sites to crawl, how often, and how many pages to fetch from each. Googlebot also slows down on purpose so it does not overload your server. If your site starts returning HTTP 500 errors or responds slowly, Google takes that as a sign to back off.
For most small and mid-sized sites, crawl capacity is not the problem. I’d only worry about it on large shops, where filter combinations create thousands of near-duplicate URLs. Our guide to crawl budget for e-commerce covers the fixes.
Googlebot Smartphone and Mobile-First Indexing
Googlebot comes in two main types: Googlebot Smartphone and Googlebot Desktop. Google mainly indexes the mobile version of most sites, so most crawl requests come from the smartphone crawler. If your mobile pages hide content that your desktop pages show, Google may never see that content.
Both crawlers follow the same robots.txt rules, so you cannot block one and allow the other.
File Size Limits Worth Knowing
In early February 2026, Google made its documentation clearer on how much of a file Googlebot fetches for Search. Googlebot crawls the first 2MB of a supported file type such as HTML, and the first 64MB of a PDF. Anything past that point is not considered for indexing.
- The limit applies to uncompressed data, not the compressed size your server sends.
- Each CSS and JavaScript file linked in the HTML is fetched separately, with its own limit.
- Google said these limits were already in place. The update just made the documentation more precise.
Few normal pages come close to 2MB of HTML. The risk is pages stuffed with huge inline scripts or base64-encoded images (images written straight into the code).
What robots.txt Does and Does Not Do
The robots.txt file sits at the root of your domain (for example example.com/robots.txt). It tells crawlers which paths they may request. A simple file looks like this:
User-agent: *
Disallow: /cart/
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xmlTwo points cause most of the confusion. This is the mistake I see most often in technical audits:
- robots.txt controls crawling, not indexing. A blocked URL can still appear in results, usually without a description, if other pages link to it.
- A
noindextag only works if Google can crawl the page. If you block a page in robots.txt and also addnoindex, Google never sees the tag. To remove a page from the index, allow crawling and usenoindex.
Sitemaps: Helpful, Not Magic
A sitemap tells Google which URLs you see as important. That is most useful for new sites, large sites and pages with few internal links. It does not force indexing.
- Include only canonical URLs that return a 200 status. Leave out redirects, error pages and
noindexpages. - One sitemap file can hold up to 50,000 URLs or 50MB uncompressed. Larger sites split them and use a sitemap index file.
- Google ignores the
priorityandchangefreqvalues. It useslastmodonly when it is accurate every time, so do not update it unless the page really changed.
Stage 2: Indexing, Rendering and Canonicalisation
Once a page is fetched, Google tries to understand it and decide whether to store it. Indexing is not guaranteed, even for pages that follow every technical rule.
Rendering: How Google Sees JavaScript Pages
Many sites build content with JavaScript. So Google renders pages using a recent version of Chrome, much like a browser would. This lets it see content that only appears after scripts run.
Rendering adds work and can add delay. Content that is in the first HTML is the safest bet. If your product descriptions, prices or internal links only appear after a script runs, check that Google really sees them. The rendered HTML view in the URL Inspection tool shows this. On JavaScript-heavy sites, it is the first thing I check.
Also make sure robots.txt does not block the CSS and JavaScript files the page needs. If Googlebot cannot load them, it may render a broken or empty page.
What Google Looks at During Indexing
Google looks at the visible text, key tags and attributes such as the <title> element and image alt text, plus images and video. It also collects signals about the page. These include its language, the country its content is relevant to, and how easy it is to use.
Structured data (code that labels facts on a page) helps Google understand things like a product’s price or an event’s date. It can also make a page eligible for rich results. Our guide to which schema markup still earns rich results covers what is worth adding.
Canonicalisation: Choosing One Version of Duplicate Pages
The same content often lives at several URLs. Think of http:// and https:// versions, URLs with tracking parameters, or a product listed under two categories.
Google groups these similar pages into a cluster and picks one as the canonical. That is the version it sees as most representative and is most likely to show in results. The others become alternates, which may still show in some cases.
You can shape that choice with consistent signals:
- A
rel="canonical"link element on each duplicate pointing to the preferred URL - 301 redirects from old or unwanted versions
- Listing only the preferred URL in your sitemap
- Internal links that always point to the preferred URL
- Serving the site over HTTPS
These are signals, not orders. If they conflict, Google may pick a different canonical from the one you set. For example, the canonical tag might say one URL while every internal link points to another. My rule is simple: make every signal point to the same URL.
Why Pages Get Crawled but Not Indexed
Google names a few common reasons a page is not indexed. The content is low quality, a robots meta rule blocks indexing, or the site design makes indexing hard. In Search Console you will often see these as statuses such as “Crawled – currently not indexed” or “Duplicate, Google chose different canonical than user”.
A few such pages is normal. When a whole section is affected, I look for a quality or duplication problem before a technical one.
Stage 3: How Google Ranks and Serves Results
When someone searches, Google looks through its index for matching pages and puts them in order by what it judges most useful. It says this is decided by computer programs and uses hundreds of factors.
Understanding the Search Comes First
Before matching pages, Google works out what the searcher means. Its language systems handle spelling mistakes, words with the same meaning, and context. Google names several AI systems in its ranking documentation:
- RankBrain links words to concepts, so a page can rank without containing the exact search words.
- Neural matching compares the concepts in a search with the concepts on a page.
- BERT helps Google understand how combinations of words change meaning, such as the role of small words like “for” or “to”.
- Passage ranking lets Google judge single sections of a long page, not just the page as a whole.
For writers, that means answering the question clearly in natural language, not repeating an exact keyword. I tell writers to write the answer the way they would say it to a customer.
Context Changes the Results
Location, language and device all shape what appears. Google’s own example is “bicycle repair shops”, which returns different results for someone in Paris than for someone in Hong Kong.
The search also decides which features appear. A product search may bring up images, while “plumber near me” brings up a local map pack.
The Ranking Systems Google Names
Google publishes a guide to its ranking systems. Besides the AI systems above, it includes:
- Link analysis systems and PageRank, which use how pages link to each other to understand what they are about and which may be most helpful.
- Freshness systems, which favour newer content where timing matters, such as news or a recent product launch.
- Reliable information systems, which aim to show trusted sources and push down low-quality content.
- Original content systems, which favour original reporting over pages that only repeat it.
- Reviews system, which rewards reviews with real analysis and first-hand research.
- Site diversity system, which usually limits the top results to two listings from the same site.
- Spam detection systems, including SpamBrain, which filter out spam and are updated regularly.
Some older names you will still read about are now retired. Panda and Penguin were folded into core ranking years ago. The helpful content system became part of the core ranking systems in March 2024. Core updates are changes to these core systems. That is why they can move rankings across many sites at once.
Quality Signals, as Google Describes Them
Google does not publish a weighted list of ranking factors. Anyone who claims to have one is guessing, and I’d be careful with them. What Google does describe is the kind of content it wants to reward: helpful, reliable and made for people, not to game rankings.
It points site owners to questions about effort, originality, expertise and trust. Its quality rater guidelines use the E-E-A-T framework (experience, expertise, authoritativeness and trustworthiness), with trust as the most important part. Raters’ scores do not directly change rankings. They help Google judge whether its systems are working. For a full breakdown, read E-E-A-T explained for your site.
Page experience also plays a part. Google says good page experience matches what its systems reward, though great content can still rank on a slower page. Our guide to Core Web Vitals targets and fixes covers the measurable side.
Where AI Overviews and AI Mode Fit In
This is the part most older explanations of Google Search miss. AI Overviews and AI Mode do not use a separate index or a separate set of rules for websites. They draw on the same crawled and indexed pages described above.
What Each One Does
- AI Overviews appear at the top of some results pages, with an AI-written summary and links to supporting sites. They only show when Google’s systems judge them useful, so many searches never show one.
- AI Mode is a separate, chat-style search experience for questions that need more reasoning, comparison or follow-up. It also returns an AI answer with supporting links.
Both can use a method Google calls “query fan-out”. Instead of running one search, the system runs several related searches across subtopics and data sources. Then it builds an answer. That is why the links in an AI answer can come from pages that would not rank for the original search on their own. Google has also tested fan-out on regular results with Web Guide, which groups links by topic instead of writing an answer.
What Changed in Early 2026
In late January 2026, Google made Gemini 3 the default model for AI Overviews around the world. At the same time, it started letting people ask a follow-up question from an AI Overview and carry on straight into a conversation in AI Mode, with the context kept.
The line between the two features is getting thinner. We go deeper on the differences in how AI Overviews and AI Mode pick their sources.
How a Page Becomes Eligible
Google’s guidance on AI features and your website is short and direct. To be shown as a supporting link, a page must be indexed and eligible to appear in Search with a snippet. There are no extra technical requirements.
- You do not need special AI files or special schema markup.
- The usual basics still apply: allow crawling, link pages internally, keep important content in text, and make structured data match what is visible on the page.
- To limit how your content appears, use the standard controls:
nosnippet,data-nosnippet,max-snippetornoindex. - Clicks and impressions from AI Overviews and AI Mode are included in Search Console’s Performance report, under the “Web” search type. They are not shown separately.
In short, if a page fails at crawling or indexing, it cannot be cited in an AI answer either. The pipeline is the same. So I’d fix the basics before spending money on any “AI SEO” tactic.
How to Check Your Page Is Crawled and Indexed
When a page is missing from Google, work through the stages in order. These steps use Google Search Console, which is free.
- Verify your site in Search Console. A domain property covers every subdomain and both
httpandhttpsversions. - Run the URL Inspection tool. Paste the full URL into the bar at the top. It tells you whether the URL is on Google and, if not, why.
- Check crawl access. In the result, look at whether crawling was allowed, when the page was last crawled, and whether the fetch worked. A robots.txt block or server error stops everything here.
- Check indexing permission and canonical. Confirm indexing is allowed (no
noindex). Then compare the user-declared canonical with the Google-selected canonical. If they differ, Google is treating another URL as the main version. - Test the live page. Use “Test live URL” and view the rendered HTML or screenshot. Confirm Google sees your main content, especially on pages that rely on JavaScript.
- Request indexing once the page is fixed. This asks Google to crawl the URL again. It does not promise indexing, and asking again and again does not speed it up.
- Review the Page indexing report. For patterns across the site, the report groups non-indexed URLs by reason. That way you can fix a whole template at once.
Google says a new crawl can take anywhere from a few days to a few weeks. Submitting a sitemap is the better route when you have many URLs to check.
Conclusion
How Google Search works comes down to a chain: discovery, crawling, rendering, indexing, picking the canonical, then ranking for each search. AI Overviews and AI Mode sit at the end of that chain and draw from the same index. When a page underperforms, find the stage where it drops out before you change anything else.
My suggested next step is to inspect your five most important URLs in Search Console. Confirm each one is crawled, indexed and using the canonical you expect. Once the basics are solid, move on to keyword research for AI search to shape content around the questions people now ask.
