Noindex in robots.txt does not work. A Noindex: line was never part of the robots.txt standard, and Google stopped honoring it on September 1, 2019, so Noindex: /private/ in your robots.txt is ignored. To keep a page out of Google, put the noindex on the page itself: a <meta name="robots" content="noindex"> tag in the HTML head, or an X-Robots-Tag: noindex HTTP header. Then leave the page crawlable. Robots.txt controls crawling, noindex controls indexing, and Google can only obey a noindex it is allowed to read.
The rest of this guide explains the difference, the trap of blocking and noindexing the same page, the exact code to use, and how to check what Google sees on any URL.
Does noindex work in robots.txt?
No. Google announced on July 2, 2019, in A note on unsupported rules in robots.txt, that it would retire all code handling unsupported and unpublished robots.txt rules, noindex included, on September 1, 2019. Before that date, Google acted on some Noindex: lines even though it had never documented the rule. Since then, it reads only four fields in robots.txt: user-agent, allow, disallow and sitemap.
The same goes for the other rules people try to put there. A Nofollow: line in robots.txt is ignored too, and Google ignores Crawl-delay:. So "robots txt noindex nofollow" setups that still circulate in old tutorials do nothing at all for Google.
A typical robots.txt noindex example, and what to do instead:
# Does nothing for Google since 2019
User-agent: *
Noindex: /thank-you/
Disallow: /thank-you/Delete both lines for that path. The first is ignored, and the second stops Google from ever seeing a real noindex. Then add the tag to the page:
<meta name="robots" content="noindex">
Meta robots vs robots.txt: what each one does
Robots.txt and noindex answer different questions. Robots.txt tells a crawler whether it may fetch a URL. Noindex tells a search engine whether the URL may appear in search results. They are not two ways of doing the same thing.
| robots.txt Disallow | noindex (meta robots tag or X-Robots-Tag) | |
|---|---|---|
| Controls | Crawling: whether Googlebot fetches the URL | Indexing: whether the URL can appear in search |
| Where it lives | One file at the root of the host | The HTML head of each page, or the HTTP response header |
| Covers | Path patterns across the whole site | One page per tag; a header rule on the server can cover many files |
| Google has to crawl the page to obey it | No | Yes |
| Can the URL still appear in Google? | Yes, without a description, when other pages link to it | No, once Google has recrawled the page and seen the rule |
| Use it for | Saving crawl time on URLs with no search value | Keeping a page out of search results |
If you want to know more about the file side of this table, our guide to robots.txt syntax and examples covers groups, wildcards and what each line does.
Why disallow and noindex on the same page backfire
Google's documentation on blocking indexing with noindex says it directly: if robots.txt blocks the page, the crawler never sees the noindex rule, and the page can still appear in search results when other pages link to it.
This is the most common way people end up with the opposite of what they wanted. They block a page in robots.txt and add noindex "to be safe". Google respects the block, never fetches the page, never reads the noindex, and keeps the URL in its index based on the links pointing to it. In search it appears with "No information is available for this page", and Search Console lists it as "Indexed, though blocked by robots.txt".
There is a second twist. When a page is blocked by robots.txt, URL Inspection always shows Indexing allowed? Yes, because Google never got far enough to find the noindex. The one field that should warn you looks fine.
How to deindex a page that robots.txt blocks
- Add noindex to the page, as a meta robots tag or an X-Robots-Tag header.
- Remove the Disallow rule that matches the page. If the change is urgent, open the robots.txt report in Search Console (Settings, then robots.txt under Crawling) and click Request a recrawl for the file.
- Test the live page. In URL Inspection, run Test live URL. You want Crawl allowed? Yes and Indexing allowed? No.
- Ask Google to crawl it. Click Request indexing. It sounds backwards, but a crawl is the only way Google sees the noindex. Without it, Google can take weeks or months to come back to a page.
- Watch the Page indexing report. Once Google recrawls the page, it moves to "Excluded by 'noindex' tag" and drops out of results.
- Leave the page crawlable. Google rechecks pages over time, and a noindex only keeps working while Google can read it.
If the page must disappear today, use the Removals tool in Search Console as well. It hides the URL for about six months while the noindex does the permanent work.
How to noindex a page so Google can see it
The meta name="robots" content="noindex" tag goes inside the <head> of the page. name="robots" addresses every search engine crawler; content lists the rules. The common versions:
<!-- Keep the page out of every search engine -->
<meta name="robots" content="noindex">
<!-- Keep it out, and do not follow the links on it -->
<meta name="robots" content="noindex, nofollow">
<!-- Keep it out of Google only -->
<meta name="googlebot" content="noindex">
A few rules from Google's robots meta tag documentation worth knowing:
noneequalsnoindex, nofollow. Either form works.followis the default.noindex, followand plainnoindexdo the same thing.- The more restrictive rule wins. If a page has a general
nofollowand a Googlebot-onlynoindex, Googlebot applies both. - Robots.txt blocks override everything here. A rule on a page Google may not crawl is never read.
For files that have no HTML head, such as PDFs, send the same rule as an HTTP header. On Nginx, for example:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex";
}In a Next.js App Router project, set it in the page's metadata and Next.js renders the meta tag for you:
export const metadata = {
robots: { index: false },
}On WordPress, use the noindex setting of your SEO plugin on the page in question, rather than a robots.txt rule.
How to check noindex and robots.txt on any URL
Three checks tell you what Google sees, from quickest to most exact:
- The free noindex checker. Paste any URL. It reads the meta robots tag (noindex or none), a googlebot meta tag, the X-Robots-Tag header, whether robots.txt blocks the URL and the HTTP status, in one pass. For the robots.txt file itself, the robots.txt checker parses every group and flags full-site blocks.
- Your browser. View the page source and search for
noindex. For the header, open the developer tools, reload, select the page in the Network tab and look forX-Robots-Tagin the response headers. - URL Inspection in Search Console. Inspect the URL, open the Page indexing details and read Crawl allowed? and Indexing allowed?. Click Test live URL to check the current version instead of the last crawl. Our URL Inspection tool guide explains every field.
Read the two URL Inspection fields together:
| Crawl allowed? | Indexing allowed? | What it means | What to do |
|---|---|---|---|
| Yes | Yes | Nothing on the page or in robots.txt blocks it. | If the page is still not indexed, the reason is crawl priority or content, not a block. |
| Yes | No | Google read a noindex rule in the meta tag or the header. | Nothing, if you meant it. If not, remove the noindex and request indexing. |
| No | Yes | robots.txt blocks crawling, so any noindex on the page is invisible to Google. | To deindex: unblock and keep the noindex. To index: unblock and remove the noindex. |
If Crawl allowed? says No but your robots.txt has no rule for that page, something in front of your server may be answering instead. CDNs and firewalls can serve their own robots.txt; our guide to robots.txt blocks caused by Cloudflare's default settings shows where to look.
Noindex vs robots.txt on 1,513 real sites
We looked at the sites connected to IndexMachine: on October 31, 2026, about 1.17 million sitemap URLs on 1,513 sites, each with its latest state in Google's URL Inspection data. We counted how many sites had at least one sitemap URL in each of the three states this post is about. Aggregate counts only, with no site or URL named; these are sites whose owners signed up for an indexing tool, so they lean toward sites with indexing trouble. Most of these sites are on free accounts, where IndexMachine has not requested anything yet, so the numbers show where sites stand before indexing is turned on.
| State in Search Console | Sites with at least one sitemap URL in it | Share of the 1,513 sites |
|---|---|---|
| Excluded by 'noindex' tag | 157 | 10.4% |
| Blocked by robots.txt | 17 | 1.1% |
| Indexed, though blocked by robots.txt | 9 | 0.6% |
Three things stand out:
- Site owners use noindex far more than robots.txt to keep pages out. 157 sites had noindexed URLs in their sitemaps, against 17 with robots.txt blocks. In URLs it was 11,843 noindexed against 5,241 blocked.
- All of these URLs were in a sitemap. A sitemap is the list of pages you want indexed. A noindexed or blocked URL in it sends Google two opposite signals. Remove it from the sitemap, or remove the rule.
- The trap is real. 9 sites had sitemap URLs that Google indexed even though robots.txt blocked them, the exact situation where a noindex behind a Disallow cannot work.
Make sure your daily requests go to pages Google can index
Noindex and robots.txt mistakes cost more than the pages they hit. Every manual request for a page that carries a noindex, or that robots.txt blocks, uses one of your limited daily Request indexing slots on a page Google will not index.
IndexMachine is built to avoid exactly that. A free account imports every site in your Search Console and shows the index status of each page in your sitemaps. With indexing turned on, its Chrome extension presses Google's own Request indexing button in your Search Console, on autopilot, every day, using your full daily Google quota on every site. Pages that cannot be indexed never use your quota, so it goes to pages that can. Pages Google already shows as indexed are skipped, every other engine gets unlimited IndexNow, and you get 404 alerts and a weekly email report. You can pay once for lifetime access or subscribe.
Find the pages Google is not indexing, and why
A free IndexMachine account imports your Search Console sites and shows every sitemap page Google has not indexed, with the reason. Turn on daily Request Indexing in your own Search Console plus unlimited IndexNow when you are ready.
FAQs
Can you noindex a page in robots.txt?
No. Google stopped supporting noindex in robots.txt on September 1, 2019, and other rules outside the standard, such as nofollow, are ignored as well. Use a meta robots noindex tag or an X-Robots-Tag header on the page, and keep the page crawlable.
Should I disallow a page in robots.txt and also noindex it?
No. If robots.txt blocks the page, Google cannot read the noindex, and the URL can stay in search based on links to it. Pick noindex to keep a page out of search, or Disallow to save crawl time on pages you do not care about. Not both on the same URL.
What does meta name robots content noindex, nofollow mean?
<meta name="robots" content="noindex, nofollow"> tells every search engine crawler not to show the page in search results and not to follow the links on it. content="none" means the same. Use noindex alone if you still want the links on the page followed.
Is noindex or robots.txt better for keeping a page out of Google?
Noindex. It is the rule built for indexing, and Google drops the page from its results once it recrawls it. Robots.txt only stops crawling, so a blocked URL can still be indexed from links. Use robots.txt for crawl control on URLs with no search value.
How long does it take for a noindex page to drop out of Google?
Until Google crawls the page again, which can take from a day to months depending on how often Google visits it. Request indexing in URL Inspection after adding the noindex to bring that crawl forward, and use the Removals tool if it has to disappear right away.