What Is Robots.txt? File Examples and Syntax

Robots.txt is a plain text file at the root of a website (example.com/robots.txt) that tells search engine crawlers which URLs they may crawl and which to leave alone. Each group of rules names a crawler with User-agent and lists paths with Disallow and Allow. Google, Bing and other well-behaved crawlers read it before they crawl anything else on the site. Robots.txt controls crawling, not indexing: it cannot remove a page from Google, and it does not protect private content. For most sites, a safe robots.txt allows everything and points to the sitemap.

Below: what robots.txt is used for in SEO, a real robots.txt file explained line by line, the syntax Google supports, robots.txt examples you can copy, how to create and test the file, and what our own data says about how often robots.txt is the problem.

What is robots.txt used for?

Robots.txt is used to manage crawler traffic. It tells search engines, and any other bot that chooses to listen, which parts of a site are not worth crawling, so they spend their visits on the pages that matter. Google's introduction to robots.txt puts it the same way: the file is mainly for keeping crawlers from overloading your site, not a way to keep a page out of Google.

In SEO, robots.txt is typically used to:

• Keep crawlers out of URLs with no search value: admin areas, carts and checkouts, internal search results, and endless filter and sort combinations.
• Save crawl time on large sites: thousands of parameter URLs can use up visits Google would otherwise spend on your new pages.
• Point crawlers to your sitemap with a Sitemap: line, which Google, Bing and the other major engines read.
• Treat bots differently: for example, keep an AI training crawler out while search engines crawl everything.

What robots.txt cannot do:

• Remove a page from Google. A blocked URL can still be indexed when other pages link to it. Google then shows it in results without a description, because it was never allowed to read the page. To keep a page out of search, let Google crawl it and add a noindex rule instead.
• Hide private content. The file is public. Anyone can open it, and listing a secret path in it advertises that path. Use a password or a login.
• Stop bad bots. Robots.txt is a request. Search engines honor it; scrapers are free to ignore it.

Robots.txt example: what the file looks like

Here is a real robots.txt example, the file indexmachine.co serves today:

User-Agent: *
Allow: /
Disallow: /dashboard
Disallow: /api/
Disallow: /checkout
Disallow: /portal

Sitemap: https://indexmachine.co/sitemap.xml

Read it top to bottom:

1. User-Agent: * starts a group that applies to every crawler.
2. Allow: / says the whole site may be crawled.
3. Each Disallow line removes one path prefix from that. /api/ covers every URL that starts with /api/, such as /api/tools/robots-txt-checker. These are app and API routes that no searcher needs.
4. Sitemap: gives the full URL of the sitemap. It does not belong to any group, so every crawler reads it.

We keep a single group on purpose. A crawler follows only one group: the one with the most specific User-agent that matches its name, and it ignores all the others. If we added a User-agent: Googlebot group with one extra rule, Googlebot would stop reading the * group and crawl /api/ and /checkout unless we repeated those lines. This is one of the most common robots.txt surprises, and it is easy to avoid: put shared rules in every group, or keep one group.

Robots.txt syntax and format

Google's robots.txt specification defines the format. The rules that matter in practice:

• Name and location. The file must be called robots.txt and sit in the top-level directory of the host: https://example.com/robots.txt. A file at example.com/blog/robots.txt is ignored.
• One file per host, protocol and port. https://example.com/robots.txt does not apply to https://shop.example.com or to http://example.com. Every subdomain needs its own file.
• Format. UTF-8 plain text, one rule per line. Lines starting with # are comments.
• Size. Google reads the first 500 KiB and ignores the rest.
• Case. Paths are case-sensitive: Disallow: /Private/ does not block /private/. Field names are not, so disallow and Disallow both work.
• Redirects. Google follows up to five redirects to reach the file, then treats it as missing.

These are the fields you can use, and whether Google supports them:

Wildcards in robots.txt

Google supports two wildcard characters:

• * matches any sequence of characters. Disallow: /*? blocks every URL that contains a question mark, which means every URL with a query string.
• $ marks the end of the URL. Disallow: /*.pdf$ blocks URLs that end in .pdf, but not /guide.pdf-download.

Which rule wins when Allow and Disallow overlap

When an Allow and a Disallow both match a URL, Google uses the most specific rule, which is the one with the longer path. If both are equally specific, Google uses the least restrictive one, which is Allow.

User-agent: *
Disallow: /shop/
Allow: /shop/sale/

Here /shop/sale/shoes may be crawled (the Allow path is longer), and /shop/cart may not.

What happens if robots.txt is missing or broken

• 404 or another 4xx error (except 429): Google treats it as if the file does not exist and crawls everything. You do not need a robots.txt file for Google to crawl your site.
• 5xx server error: for the first 12 hours, Google stops crawling the whole site and keeps retrying the file. After that it uses the last good copy for up to 30 days. A robots.txt that errors is worse than none at all.

Google also caches your robots.txt for up to 24 hours, so a change can take a day to apply.

Robots.txt examples you can copy

Most sites need one of these. Replace example.com with your domain.

Robots.txt allow everything

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Robots.txt disallow all crawlers

User-agent: *
Disallow: /

Use this only on a staging or test site, and remove it before launch. A Disallow: / left in place is the classic reason a newly launched site never shows up in Google.

Block one folder and one bot

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

GPTBot is the crawler OpenAI uses to collect training data. It follows only its own group, so it is blocked everywhere, while Googlebot, Bingbot and every other crawler follow the * group and skip only the cart and internal search pages.

The WordPress default

WordPress serves this file when no physical robots.txt exists:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

How to create a robots.txt file

Creating a robots.txt file takes a text editor and five minutes:

1. Check whether you already have one. Open https://yourdomain.com/robots.txt in a browser. WordPress, Shopify, Wix, Squarespace and most other platforms create one for you.
2. Write the rules in a plain text editor, not a word processor. Start from "allow everything" and add a Disallow line only for paths you are sure have no search value.
3. Add your sitemap as a full URL: Sitemap: https://yourdomain.com/sitemap.xml.
4. Save the file as robots.txt in UTF-8 and upload it to the root of the host, next to your homepage. Repeat for every subdomain that needs rules.
5. Test it. Open the URL again and confirm you see your rules. Then run it through the free robots.txt checker, which parses every user-agent group and flags full-site blocks, crawl-delay lines, file size problems and duplicate rules.
6. Check Google's copy in Search Console. Go to Settings, find robots.txt under Crawling and click Open report. It lists the robots.txt files Google found for your top hosts, when Google last checked each one, the fetch status (Fetched, Not Fetched - Not found (404) or Not Fetched - Any other reason) and any parsing issues.
7. Ask Google to read the new file. In the same report, open the more settings menu (three dots) next to the file and click Request a recrawl. Without that, Google picks up the change within about a day.
8. Spot-check an important page. Paste its URL into URL Inspection: Crawl allowed? should say Yes. Our URL Inspection tool guide explains the other fields on that screen.

Platform notes:

• WordPress: the default file is virtual. Edit it in Yoast SEO (Yoast SEO, Tools, File editor) or Rank Math (General Settings, Edit robots.txt), or upload a physical robots.txt, which replaces the virtual one.
• Shopify: the default file is generated. To change it, add a robots.txt.liquid template to your theme.
• Next.js: return the rules from app/robots.ts, as we do, or put a static file in public/.

Common robots.txt mistakes

Most robots.txt problems come from a handful of mistakes, and Search Console usually shows the symptom before you notice the cause:

If Search Console says pages are blocked by robots.txt and your own file looks fine, something in front of your server may be answering for it. CDNs and firewalls can serve their own robots.txt or block Googlebot outright; our guide to robots.txt blocks caused by Cloudflare's default settings walks through that case.

How often is robots.txt the reason pages are not indexed?

Rarely, but when it is, it hits many pages at once. Here is what our own data shows.

Our own site. We run URL Inspection on every URL in the indexmachine.co sitemap. On October 28, 2026, all 71 URLs showed Crawl allowed? Yes, and 68 were indexed. The 3 that were not had other reasons: 2 were "Crawled - currently not indexed" and 1 was a soft 404. A re-run on October 31 gave the same crawl result for all 71.

The 1,513 sites connected to IndexMachine. On October 31, 2026, 5,241 of their roughly 1.17 million sitemap URLs (0.4%) were "Blocked by robots.txt", on 17 sites (1.1% of sites). On most of those sites it was a single URL, and one site accounted for most of the 5,241: robots.txt mistakes are rare, but one line can block a whole folder or a whole site. Most of these sites are on free accounts, where IndexMachine has not requested anything yet, so the numbers show where sites stand before indexing is turned on. Aggregate counts only, with no site or URL named.

What people ask Google. Between August 1 and October 29, 2026, Google showed our pages for 124 different searches containing "robots", 21,038 times. 20,539 of those impressions went to our free robots.txt checker, for searches like "robots txt checker", "check robots.txt" and "robots.txt validator". The common robots.txt question is not "how do I write one" but "is mine right?", which is why testing is part of creating the file, not an extra.

Robots.txt is open, now get the pages indexed

A clean robots.txt lets Google crawl your pages. It does not make Google index them, and for most sites the gap is large: crawlable pages sit as "Discovered - currently not indexed" or "URL is unknown to Google" for weeks.

That gap is what IndexMachine closes. A free account imports every site in your Search Console and shows which sitemap pages Google has not indexed and why. With indexing turned on, its Chrome extension presses Google's own Request indexing button in your Search Console, on autopilot, every day, using your full daily Google quota on every site. Pages Google already shows as indexed are skipped. Pages Google would not index anyway never use your quota. Every other engine gets unlimited IndexNow submissions, and you get 404 alerts, daily reports in the dashboard and a weekly email report. You can pay once for lifetime access or subscribe.

FAQs

What is a robots.txt file?

A robots.txt file is a plain text file at the root of a website that tells crawlers which URLs they may crawl. It uses User-agent lines to name crawlers and Disallow and Allow lines to list paths. Search engines read it before crawling the site.

What is robots.txt in SEO?

In SEO, robots.txt is how you keep search engine crawlers away from URLs with no search value (admin pages, carts, internal search, filter combinations) so they spend their time on the pages you want found. It also points crawlers to your sitemap. It does not decide what gets indexed or ranked.

Do I need a robots.txt file?

No. If the file returns a 404, Google crawls everything it can find. Small sites with nothing to hide from crawlers do fine without one. A two-line file that allows everything and lists your sitemap is still useful, because every engine finds the sitemap there.

Does robots.txt Disallow remove a page from Google?

No. Disallow stops crawling, not indexing. A blocked URL that other pages link to can stay in Google's index, shown without a description. To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header, or return 404 or 410.

How long does Google take to see a robots.txt change?

Up to about 24 hours, because Google caches the file. To speed it up, open the robots.txt report in Search Console (Settings, then robots.txt) and use Request a recrawl on the file.