Googlebot Simulator

What the Googlebot simulator shows

It requests your URL with the user agent of Googlebot Smartphone, the crawler Google uses for almost every site since mobile-first indexing, and shows what comes back:

• The HTTP status and the redirect chain, hop by hop, so you see whether Google lands on a 200, a 404 or a loop.
• The response headers, including X-Robots-Tag, Link canonicals, caching and the server.
• robots.txt for Googlebot: whether the URL may be crawled and which rule decides, for the URL and for where it redirects.
• Index signals: meta robots, meta googlebot, the canonical tag, title, meta description and H1. To see the title and description in a Google preview, use the title tag checker.
• The text and links in the HTML, the way a crawler reads them before any JavaScript runs.

Then it fetches the same URL as Chrome on Android, the same kind of phone Googlebot pretends to be, and flags every difference. A different status, redirect, canonical or much less text for Googlebot is how cloaking, bot protection and broken dynamic rendering show up. A short AI summary sums up the problems and the fixes.

Fetch as Google is now URL Inspection

Google retired the old Fetch as Google tool together with the old Search Console. Its replacement is URL Inspection: paste a URL of a site you have verified, run the live test and Google shows what it fetched and rendered. That is the final word for your own pages, and you should use it to confirm anything you find here.

This simulator fills the gaps. It works on any public URL, without a Search Console account, including a client's site before you have access or a competitor's page you want to compare. It also shows the raw response headers, every link and the browser comparison, which URL Inspection does not.

How the simulator differs from the real Googlebot

Two differences matter. First, the request comes from our servers, not from Google's IP addresses. Sites that verify Googlebot by IP or reverse DNS, as Cloudflare and many security plugins do, may refuse it, and the tool tells you when that is the likely cause. Second, it does not run JavaScript. Google renders pages in a second pass with a recent Chrome, so content built by JavaScript can still get indexed, just later and less reliably. The text view here is what Google gets on its first pass, and all most other crawlers ever see.

Reading the results

To validate your whole robots.txt file, rule by rule, use the robots.txt checker. To see every place a noindex can hide, including the HTTP header, use the noindex checker. To trace every hop of a redirect with its status code, use the redirect checker.

Googlebot can read the page, but it is still not indexed

Crawlable is not the same as indexed. When the simulator shows a clean 200 with no noindex and a self canonical, Google has simply not chosen the page yet, which Search Console reports as Discovered or Crawled, currently not indexed. The why is my page not indexed checker looks at content, links and sitemaps for the reason, and the path out is Request Indexing in Search Console.

IndexMachine does that part on autopilot. Its Chrome extension clicks Google's own Request Indexing in your Search Console every day for every page Google hasn't indexed, with your full daily quota on every site, and sends every new URL to Bing, Yandex, Seznam and Naver through unlimited IndexNow. No link farms. Of 1,841 pages IndexMachine requested, 1,418 (77%) were on Google, typically by our first check about 3 days after the request. Sign up free to see which of your pages Google hasn't indexed, or compare plans on the pricing page.