A search engine index is the database of web pages a search engine has found, read and stored, so it can answer a search in a fraction of a second. When you search Google, you are not searching the live web. You are searching Google's index, which Google says covers hundreds of billions of webpages and works like the index at the back of a book, with an entry for every word seen on every page it indexes. Indexing is the step that adds a page to that database, and a page that is not in the index cannot appear in search results.
This guide explains what is inside the index, how a page gets into it, what Googlebot does, how Google decides what to keep, and how to check and get your own pages indexed. It also shows what 1.24 million sitemap URLs on 1,513 sites reveal: Google left about 1 in 4 of the URLs it crawled out of its index.
What is an index in a search engine?
An index in a search engine is a lookup table from words to pages. Instead of reading pages one by one when you search, the engine flips the data around ahead of time: for every word it has seen, it keeps a list of the pages that contain it. Computer scientists call this an inverted index, and it is why a search over hundreds of billions of pages can come back almost instantly.
Take three guides on this blog: our soft 404 guide, our sitemap guide and our robots.txt guide. An index of just those three pages would hold entries like these:
| Word | Pages that contain it |
|---|---|
| sitemap | Soft 404 guide, sitemap guide, robots.txt guide |
| canonical | Soft 404 guide, sitemap guide |
| robots.txt | Sitemap guide, robots.txt guide |
| lastmod | Sitemap guide |
| disallow | Robots.txt guide |
A search for robots.txt lastmod never opens the pages. The engine reads two short lists, "robots.txt" (two pages) and "lastmod" (one page), keeps the page that is on both, the sitemap guide, and only then ranks what is left.
Google's founders described this structure in their 1998 paper, The Anatomy of a Large-Scale Hypertextual Web Search Engine: for every word, a list of the documents that contain it, with where and how the word appears in each. They built it in two passes: first a forward index that lists the words on each page, then a sort by word that turns it into the inverted index. Today's index holds much more than words. Google notes key signals "from keywords to website freshness", and its guide to how Search works names others, such as the page's language, the country the content is local to and its usability. The principle is the same: a page with no entries in the index cannot be reached by any search.
What is Google indexing? Crawl, render, index, serve
Google indexing means Google has crawled a page, processed it and stored it in its index, so the page is eligible to appear in search results. In SEO, "indexed" is the line between a page that can get search traffic and one that cannot. The same line holds for Google's AI answers: to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Google Search with a snippet, and there are no extra technical requirements. Google describes three stages, crawling, indexing and serving, and says that not all pages make it through each stage. Two steps inside them are worth naming on their own: discovery, before the crawl, and rendering, between the crawl and indexing.
| Step | What Google does | If the page stops here, Search Console shows |
|---|---|---|
| 1. Discover | Finds the URL in a link from a page it already knows or in a sitemap, or when you request indexing for it, and adds it to its list of known pages. | URL is unknown to Google |
| 2. Crawl | Googlebot downloads the page, if robots.txt allows it and the server answers. | Discovered - currently not indexed (known, not crawled yet); Blocked by robots.txt; Server error (5xx) |
| 3. Render | Runs the page's JavaScript in a recent version of Chrome to see what a visitor would see. | No state of its own. A failed render can show up as Soft 404 or as content missing from the index. |
| 4. Index | Reads the text, key tags such as the title and alt attributes, images and videos, groups duplicate pages, picks one canonical URL and decides whether to store the page. | Crawled - currently not indexed; Duplicate, Google chose different canonical than user; Excluded by 'noindex' tag |
| 5. Serve | Ranks indexed pages for each search, using hundreds of factors such as the searcher's location, language and device. | URL is on Google, but few or no impressions in the Performance report |
The steps people mix up most are crawling and indexing. Crawling is Googlebot fetching a page. Indexing is Google deciding to keep it. A page can be crawled again and again and never be indexed, and Google says so directly in its crawl budget guide: "not every page that is crawled will necessarily be indexed". Rendering is where JavaScript sites lose content without noticing; our guide to how JavaScript rendering affects indexing covers what breaks there and how to spot it.
What is Googlebot?
Googlebot is Google's web crawler, the program that fetches pages for Google Search. Google uses the name for two crawlers:
- Googlebot Smartphone simulates a user on a mobile device. It makes the majority of Googlebot's requests, because Google uses the mobile version of a site's content for indexing and ranking (mobile-first indexing).
- Googlebot Desktop simulates a user on a desktop computer and makes the rest.
Both obey the same Googlebot token in robots.txt, so you cannot allow one and block the other. URL Inspection shows which one fetched a page under Crawled as. On our own site it was Googlebot smartphone for all 92 of our URLs that Google had crawled, in our checks on October 31 and November 11, 2026.
How Googlebot behaves, from Google's documentation:
- It reads the first 2MB of a page. Googlebot fetches the first 2MB of a supported file type (64MB for a PDF), counted on the uncompressed file, and only that part goes on to indexing. CSS and JavaScript files are fetched separately, each under the same limit.
- It tries not to overload your server. For most sites, Googlebot should not access the site more than once every few seconds on average, and server errors such as HTTP 500 tell it to slow down.
- It follows real links. Google can generally crawl a link only if it is an
<a>element with anhrefattribute, so pages reachable only through buttons or script handlers may not be found. - Real Googlebot can be verified. Anyone can copy its user agent, so check the IP: for Googlebot, a reverse DNS lookup should return a name ending in googlebot.com (Google's other crawlers and fetchers use google.com or googleusercontent.com), and a forward lookup of that name should return the same IP.
For the whole site, the Crawl stats report (under Settings in Search Console) shows Googlebot's requests, download size and response times, split by Googlebot type and by purpose (Discovery for new URLs, Refresh for recrawls). To see a single page the way Googlebot fetches it (status code, redirects, robots.txt and meta robots rules, canonical, text and links), run it through the free Googlebot simulator.
Does Googlebot index pages?
Strictly, no. Googlebot crawls: it fetches pages and passes them on. Analyzing the page, grouping duplicates and deciding whether to store it is indexing, a separate step with its own rules. "Googlebot indexed my page" is everyday shorthand for "Google crawled and indexed it", which is fine as long as you remember the two can come apart. A crawl is Googlebot's job; a place in the index is Google's decision.
How does Googlebot find pages and use your sitemap?
Googlebot finds new pages in three ways:
- Links. The main one. Google extracts links from pages it already knows and adds the new URLs to its list. It uses links both to find new pages and as a signal of relevance, so link every new page from pages Google already visits, such as your homepage or a hub page.
- Sitemaps. A sitemap lists the URLs you want crawled, and Googlebot treats it as a hint, not an order. Google calls submitting a sitemap "merely a hint" that does not even guarantee Google will download it. It trusts your
lastmoddates only when they are consistently accurate (how to set lastmod so Google uses it), and it ignorespriorityandchangefreq. - Request Indexing. The URL Inspection tool in Search Console lets you ask Google to crawl one URL. There is a daily limit, and Google says crawling can take anywhere from a few days to a few weeks; see how long Google takes to index new pages for typical timelines.
So does Googlebot crawl a sitemap from top to bottom? No. Google decides which sites to crawl, how often and how many pages to fetch from each with its own algorithms, based on how much your server can handle and how much Google wants your pages. Together those make up your site's crawl budget. Being in a sitemap does not mean Google has visited a URL, or even heard of it: in our data below, 43.4% of sitemap URLs were still unknown to Google (26% on the median site), and on our own site 2 new pages still showed as URL is unknown to Google 11 days after launch. For what belongs in the file in the first place, see what a sitemap is and what to put in it.
How does Google decide what to index?
Google indexes the pages it judges worth showing, and only one version of each. After the crawl, Google's crawl budget guide says, each page "must be evaluated, consolidated, and assessed" to decide whether it belongs in the index. Four checks settle most cases:
- Duplicates and canonicalization. Google groups pages with the same or very similar content and picks one as the canonical, the URL that may be shown in results. The others are not indexed on their own. Your
rel="canonical"tag, redirects, sitemap and HTTPS are signals Google weighs, but Google makes the final choice. - Quality. Google lists low-quality content as a common reason a page is not indexed. A page that adds little to what the index already holds, on your site or elsewhere, can be crawled and left out. In Search Console, such pages usually show as Crawled - currently not indexed.
- Your own rules. A
noindexrule in a robots meta tag or anX-Robots-Tagheader keeps a page out on purpose. - Errors. A page that returns a 404, a server error or an empty soft 404 gives Google nothing to index.
None of these decisions is final. Google recrawls pages it already knows (the Refresh crawls in Crawl stats) and reindexes them based on several criteria, including how often it thinks they change, so a page left out can get in later, and a page that was indexed can drop out of the index. Our guide on what to do when Google deindexes a page covers how to tell why it left and how to get it back.
Google is also clear that you should not expect all URLs on your site to be indexed, only the canonical pages. Every healthy site has some URLs outside the index: redirects, duplicates, filtered views. The ones to worry about are the pages you want people to find.
What we found across 1,513 sites: crawled is not indexed
To see where pages actually stop, we looked at the sites connected to IndexMachine: the 1,513 sites with at least one sitemap URL that had a URL Inspection result, as of November 11, 2026. That is 1,239,239 sitemap URLs, each with its most recent URL Inspection state, so some are days old and some older. Aggregate counts only, with no site or URL named. Because their owners came to us for help with indexing, the sites probably lean toward indexing trouble. Most of them had no page requested through IndexMachine, so the numbers mostly show what Google does on its own.
| Where the sitemap URL stood | Sitemap URLs | Share |
|---|---|---|
| Not discovered: URL is unknown to Google | 537,340 | 43.4% |
| Known to Google, not crawled (including 5,050 blocked by robots.txt) | 240,545 | 19.4% |
| Crawled, not indexed | 112,808 | 9.1% |
| Indexed | 348,546 | 28.1% |
| All inspected sitemap URLs | 1,239,239 | 100% |
So 56.6% of the sitemap URLs were known to Google, 37.2% had been crawled and 28.1% were indexed. Of the roughly 461,000 URLs Google had crawled, 112,808 (24.5%) were not indexed: about 1 in 4. This is not just a big-site effect. Of the 817 sites with at least 10 crawled sitemap URLs, 648 (79%) had at least one that Google crawled and left out, and on the median site it was 10% of its crawled URLs.
Here is why Google left those 112,808 crawled URLs out:
| Why the crawled URL was not indexed | State in URL Inspection | Crawled URLs | Share |
|---|---|---|---|
| Google chose not to index it | Crawled - currently not indexed | 68,532 | 60.8% |
| A duplicate: Google indexed another URL instead | Duplicate, Google chose different canonical than user; Alternate page with proper canonical tag; Duplicate without user-selected canonical | 15,750 | 14.0% |
| The site asked Google not to index it | Excluded by 'noindex' tag | 13,864 | 12.3% |
| It redirects | Page with redirect, Redirect error | 9,568 | 8.5% |
| An error | Not found (404), Soft 404, Server error (5xx), 403 and other 4xx errors | 4,985 | 4.4% |
| Other | Discovered - currently not indexed, with an older crawl on record | 109 | 0.1% |
What it shows:
- Most crawled pages that stay out are Google's choice, not a bug. 60.8% were Crawled - currently not indexed: fetched without an error, then left out, at least for now. There is no tag to fix, only a page to make more useful or better linked.
- Duplicates come second. 14.0% lost out to another URL with the same content, which is canonicalization at work.
- Technical faults are the minority. Errors and redirects together were 12.9%, and the sites' own noindex rules another 12.3%.
What happened to our own new pages
We see the same steps on our own site. On October 31, 2026 we launched 21 new pages on indexmachine.co (free tools, product pages and hub pages), and on November 2 we started publishing one blog post a day. On November 11 we ran URL Inspection on every new URL that had been live for at least 7 days.
| Our URLs | Live since | Checked | Indexed | Not indexed yet |
|---|---|---|---|---|
| 71 URLs already in our sitemap | Before Oct 31 | Oct 31 | 68 (96%) | 3 Crawled - currently not indexed |
| 21 pages launched together | Oct 31 | Nov 11 | 15 (71%) | 3 Crawled - currently not indexed, 1 Discovered - currently not indexed, 2 URL is unknown to Google |
| First 3 daily blog posts | Nov 2 to 4 | Nov 11 | 3 (100%) | None |
By November 11, when they had been live 7 to 11 days, 18 of the 24 new URLs (75%) were on Google, and for every one of them Google chose our own URL as the canonical. The 6 that were not sat at three different steps: 2 still showed as URL is unknown to Google, 1 Google knew but had not crawled, and 3 it crawled and chose not to index. Two of those 3 are hub pages that mostly list and summarize our other pages, which is the likely reason: most of what they say was already in the index. And one of the 3 older pages left out on October 31 was the Googlebot guide this article replaces: Google last crawled it on August 2, and it was still not indexed.
Indexing vs ranking: what is the difference?
Indexing decides whether a page can appear in Google at all. Ranking decides where it appears for a given search. An indexed page is a candidate, and ranking is the contest between candidates, run again for every query with hundreds of factors, such as the searcher's location, language and device.
That is why a page can show URL is on Google in Search Console and still get no visitors. Google lists three reasons an indexed page does not appear: the content is irrelevant to what people search, its quality is low, or robots meta rules prevent serving. Being indexed is necessary, not sufficient. Neither step can be bought, either: Google says no one can pay it to crawl a site more often or to rank it higher.
What is page indexing in Google Search Console?
Page indexing in Google Search Console is Google's own record of which of your pages are in its index and why the others are not. You read it in two places. The Page indexing report, under Indexing, then Pages, is the site-wide view: indexed and not indexed counts over time, with every reason Google gives listed under Why pages aren't indexed. URL Inspection, the search bar at the top of Search Console, is the view for one page: URL is on Google, or URL is not on Google with the reason, plus the last crawl and the canonical Google chose, all covered field by field in our guide to the URL Inspection tool.
Both are more reliable than a site: search on Google, which Google says does not necessarily return every indexed URL: a result there means a page is indexed, but no result proves little.
How do Bing and other search engines index pages?
Every search engine runs its own crawler and keeps its own index, so a page can be on Google and missing from Bing, or the other way around. Bing reads the sitemaps you submit in Bing Webmaster Tools, and it also accepts IndexNow, a free protocol that lets a site tell search engines as soon as a URL is added, updated or deleted, instead of waiting for the next crawl. One submission is shared with every IndexNow engine, including Bing, Yandex, Seznam.cz, Naver and Yep. Google is not on that list, so Google still needs its own crawl.
Bing's index also reaches beyond Bing.com. Bing's AI Performance report (February 2026) covers citations in Microsoft Copilot and in Bing's AI-generated summaries, and Bing says IndexNow helps AI systems reference the most current version of a page.
How do you get a page indexed?
Make the page worth indexing and easy to reach, then ask:
- Make it indexable. It returns 200, robots.txt allows it, it has no
noindex, and its canonical points to itself. - Link to it from pages Google already crawls: your homepage, a hub page or a related post.
- List it in your sitemap, with an accurate
lastmod. - Ask Google to crawl it. In URL Inspection, click Test live URL, then Request indexing. Requesting the same URL again will not get it crawled faster.
- Tell the other engines with IndexNow.
For a brand-new site, the whole sequence, from verifying Search Console to earning your first links, is in our guide on how to get your website on Google. None of it forces Google: requesting a crawl does not guarantee inclusion, and the 24.5% of crawled URLs left out above is why the page itself has to earn its place.
Step 4 is the one that does not scale by hand: one URL per request, a daily cap on each property, and another round for every new page and every page that later drops out of the index. IndexMachine takes it over with Google's own Request Indexing in your Search Console, on autopilot. Every day, the IndexMachine Chrome extension makes the request a site owner would make by hand, using your full daily Google quota on every site and none of it on pages Google already shows as indexed. Every page it requests stays tracked until Google has it, and Bing, Yandex, Seznam, Naver and the other IndexNow engines hear about new sitemap pages through unlimited IndexNow. It works only on your own verified sites: no link farms, no third-party URLs. Of 1,841 pages requested, 1,418 (77%) were on Google, typically in about 3 days by our first check. See pricing: pay once for lifetime access, or subscribe.
See which of your pages are in Google's index
Sign in with Google and a free IndexMachine account shows which of your sitemap pages are in Google's index and which are not, one page at a time. Then switch on daily Request Indexing in your own Search Console, with unlimited IndexNow for the other engines.
FAQs
What is website indexing?
Website indexing is a search engine storing a site's pages in its index after crawling them, so they can appear in search results. When people say a website is indexed, they usually mean some of its pages are. Check the pages that matter one by one in URL Inspection, because most sites have some pages indexed and others not.
Does Google index every page it crawls?
No. Google crawls many pages it decides not to keep. Of the roughly 461,000 sitemap URLs Google had crawled on the 1,513 sites in our data (November 11, 2026), 24.5% were not indexed, most of them Crawled - currently not indexed. Google's own documentation says the same: not every page that is crawled will necessarily be indexed.
Can Google index a page without crawling it?
Yes. The usual case is a page that robots.txt blocks but other pages link to: Google can index the URL without visiting the page and show it in results without a description. Search Console reports it as Indexed, though blocked by robots.txt; in our data, 337 sitemap URLs on 8 sites were in that state. To keep a page out of Google, allow crawling and add noindex instead, so Google can read the rule.
What is the Google Indexing API?
It is an API that lets a site tell Google directly when certain pages are added or removed, so Google can schedule a fresh crawl. Google limits it to two page types: job postings and livestream videos (pages with JobPosting markup or a BroadcastEvent embedded in a VideoObject), and does not support it for anything else. The Search Console API has no indexing method either, as our guide to the Search Console API explains, so for an ordinary page you still ask Google with a sitemap or with Request indexing in URL Inspection.