Blocked by robots.txt on Cloudflare: Fix It

Getting a "Blocked by robots.txt" error in Google Search Console but didn't configure it yourself? This issue often stems from Cloudflare's default settings, which can unintentionally block Googlebot or change the robots.txt file Google receives. When this happens, your pages, CSS, or JavaScript files may not appear in search results, harming your site's visibility.

Here's the quick fix:

• Check for crawl errors in Google Search Console, especially under "Indexing > Pages."
• Review Cloudflare settings like Bot Fight Mode, Super Bot Fight Mode, AI bot policies, and WAF rules. These can block legitimate bots.
• Verify robots.txt behavior using Cloudflare's security events and Google Search Console's URL Inspection tool.
• Adjust Cloudflare settings to allow verified bots like Googlebot and Bingbot by modifying firewall and bot rules.
• Update your robots.txt file to prevent errors caused by Cloudflare's internal paths (e.g., disallow /cdn-cgi/).

You'll also want to confirm fixes by testing live URLs in Google Search Console and monitoring crawl stats for improvements. This ensures search engines can access your site without unnecessary blocks. Let's dive into the details to resolve this issue and keep your content visible.

Finding the Source of 'Blocked by robots.txt' Errors

Check Crawl Errors in Google Search Console

Start by logging into Google Search Console and navigating to Indexing > Pages. This is where you'll find details on URL errors flagged as "Blocked by robots.txt" or other crawl anomalies. Look for patterns in these errors to determine what might be causing the issue.

Use the URL Inspection Tool for a closer look. Enter one of the problematic URLs and click "Test Live URL." The Crawl allowed? line tells you whether Google read a robots.txt rule that blocks the page, and Page fetch tells you whether the page itself came back.

It helps to know how Google treats a robots.txt file it cannot read, because a firewall is often the reason. Google's robots.txt specification says:

• A 4xx response (except 429) is treated "as if a valid robots.txt file didn't exist", so Google assumes there are no crawl restrictions. A Cloudflare 403 on the robots.txt file alone does not block your site.
• A 5xx response, a 429, a timeout or a broken connection is treated as a server error: "For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file." After that, Google uses the last good version for up to 30 days.

So pay close attention to the other errors in your crawl reports. If you notice "Blocked due to access forbidden (403)" or "Server error (5xx)" next to the robots.txt warnings, a firewall rule is the likely cause rather than robots.txt itself. According to Don Jiang from Guangsuan Technology, "it could be due to firewall blocks".

Check the robots.txt report in Google Search Console (under Settings). If the status reads "Not fetched" (for any reason other than a 404), Cloudflare may be blocking access to your robots.txt instructions. Also open the version Google fetched and compare it with the file on your server: Cloudflare can add lines of its own (more on that below). To confirm a firewall block, compare the timestamps of these crawl errors with the events in Cloudflare's Security > Analytics (the Events tab). Look for entries where the User-Agent is "Googlebot" and check if Cloudflare issued a "Block", "Managed Challenge" or "JS Challenge".

Export the crawl error data from GSC as a .csv file. This will be helpful if you need to involve Cloudflare support to investigate specific IP blocks or rule triggers.

Next, review your Cloudflare DNS proxy settings to determine whether the issue is tied to their security layers.

Verify DNS Proxy Settings with Cloudflare

After analyzing crawl errors, it's essential to check your DNS proxy settings in Cloudflare. This step helps identify whether the issue stems from Cloudflare or your origin server.

Run the following command in a terminal: dig +short SOA www.example.com (replace "www.example.com" with your domain). If the output includes cloudflare.com, you're using a Full setup. If it shows cdn.cloudflare.net, you're on a Partial (CNAME) setup.

To diagnose further, temporarily switch your DNS records from "Proxied" (orange cloud) to "DNS Only" (gray cloud). If the crawl errors disappear after this change, it confirms that Cloudflare's security settings are blocking the crawlers.

You can bypass Cloudflare entirely to test your origin server by running: curl --header "Host: www.example.com" http://[Origin-IP]/robots.txt. If your origin server responds with HTTP 200 OK and serves the correct robots.txt file, but Google Search Console still shows blocks, then Cloudflare is the source of the problem.

In Cloudflare's security events, filter for entries related to "Googlebot" or "/robots.txt". As Cloudflare Community MVP cbrandt suggests: "Check your Firewall Events on Cloudflare's dashboard... and use the filter to find out if there are events related to path /robots.txt. Take note of what service is blocking or challenging Google's crawlers". These logs will reveal specific Rule IDs and services (such as WAF or Bot Fight Mode) that are interfering with Googlebot.

Lastly, open Cloudflare's AI Crawl Control and go to the Directives tab. Its Robots.txt availability table shows how often each hostname's robots.txt was requested, split into successful and unsuccessful requests, with the HTTP status. Cloudflare's own guidance is that a high number of "Unsuccessful" requests (status codes 400 or above) suggests crawlers are having trouble reaching your robots.txt, so check WAF rules and other security settings.

Fixing Cloudflare Settings to Remove Robots.txt Blocks

Turn Off Anti-Bot Features and Super Bot Fight Mode

To address the blocks identified earlier, you'll need to tweak your Cloudflare settings. In the Cloudflare dashboard, go to Security > Settings, filter by Bot traffic, and open Super Bot Fight Mode. At a minimum, set "Verified bots" to Allow: Googlebot and Bingbot are verified bots. To turn the feature off completely, Cloudflare's documented steps are to set "Definitely automated traffic," "Likely automated traffic," and "Verified bots" to Allow, and turn off "Static resource protection," "JavaScript detections," and "Optimize for WordPress."

If you're on a Pro, Business, or Enterprise plan, you'll have more control over bot management through Super Bot Fight Mode, including WAF custom rules with the Skip action for parts of your site where you want bot traffic. Users on the Free plan are limited to Bot Fight Mode, which is stricter and less customizable. It's important to note that, per Cloudflare's Bot Fight Mode docs, "you cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules." It does not trigger when an IP Access rule matches the request first, so that is the way to exempt specific traffic; otherwise, turn Bot Fight Mode off in the same Bot traffic settings.

Another setting to watch out for is AI bot blocking. On September 15, 2026, Cloudflare replaced its single "Block AI bots" switch with AI bot policies for three kinds of bots: Search, Agent and Training. New domains now block Training and Agent bots on pages that display ads, while "Search will remain allowed." Each blocking option "will block Verified bots classified with that behavior", so open Security Settings > Configure AI bot policies and make sure Search is set to Allow unless you really want search crawlers out.

Once you've adjusted these settings to allow legitimate traffic, the next step is to configure firewall rules for verified search engine crawlers.

Adjust Firewall and IP Rules for Search Engine Crawlers

After fine-tuning your bot settings, you'll want to refine your firewall rules to ensure verified crawlers, like Googlebot and Bingbot, can bypass Cloudflare's restrictions. Create a WAF custom rule with the expression cf.client.bot (Cloudflare's "Known bots" field) and an Allow or Skip action. This ensures that requests from known, verified crawlers are permitted.

To make sure these rules take effect, place your "Allow" or "Skip" rules for crawlers at the top of your rule list. This ensures they're prioritized before any broader "Block" or "Challenge" rules. Cloudflare's crawl error troubleshooting guide also says not to block Google crawler IP addresses with custom rules or IP Access rules, and to make sure rate limiting rules do not apply to the Google crawler. Instead of trusting individual IP addresses, which change, verify Google's crawlers the way Google recommends: with a reverse DNS lookup that ends in googlebot.com, google.com or googleusercontent.com, or against Google's published IP ranges.

As digital marketing expert Raman Singh advises: "A safe whitelist uses verification signals and narrow scope. A risky whitelist uses only a user agent match or a broad IP allow". Avoid creating rules based solely on the User-Agent string, as it can be easily spoofed by attackers.

Update Robots.txt for Better Crawler Access

To prevent crawl errors related to Cloudflare, you'll need to update your robots.txt file. Add the directive Disallow: /cdn-cgi/ to your file. Cloudflare itself recommends this, stating: "Do not allow crawling of files in the /cdn-cgi/ directory. This path is used internally by Cloudflare and Google encounters errors when crawling it". Cloudflare adds that errors for cdn-cgi "do not impact site rankings", so this is housekeeping, not an emergency.

Next, check whether Cloudflare is writing part of your robots.txt. Its managed robots.txt setting (Security Settings, filter by Bot traffic, Configure AI bot policies, then Enable Bot Preference Sync) prepends Cloudflare's managed directives and its Content Signals Policy to your existing file when at least one AI bot policy is set to block. The managed block disallows AI crawlers such as GPTBot, CCBot, ClaudeBot and Google-Extended, while a User-Agent: * group with Allow: / keeps search engines in. If you have no robots.txt at all, Cloudflare creates one for you, and Free plan domains without a robots.txt of their own show the Content Signals Policy to crawlers. That explains a robots.txt you never wrote; it does not, by itself, block Googlebot. Cloudflare also warns that Search Console "may occasionally report Syntax not understood for Content Signals", and says it has seen no impact on crawling or SEO from those reports.

The free robots.txt checker fetches your live robots.txt the way a crawler gets it and lists every user-agent group and rule in it, so you can see exactly what Cloudflare added.

For ongoing monitoring, check the Directives tab in AI Crawl Control. This will show you how bots are interacting with your file. If the dashboard displays 404 Not Found, you'll need to create a robots.txt file. If you notice HTTP status codes of 400 or higher, review your WAF rules or other security settings that might be blocking access to the file.

Checking and Confirming the Fix

Track Indexing Status in Google Search Console

Once you've made adjustments in Cloudflare, it's essential to confirm that search engines can access your site again. Start by using Google Search Console's URL Inspection tool on a page that was previously blocked. Click on "Test Live URL" and check for the results to display "Crawl allowed? Yes" and "Page fetch: Successful". Also, ensure that all critical elements (like CSS, JavaScript, and images) are loading properly.

Next, review the Crawl Stats in Google Search Console. Look for a decrease in 403 errors and an increase in 200 responses, which indicate that crawling has recovered. In the Page indexing report, you should see fewer URLs listed under "Blocked by robots.txt" as pages become accessible to search engines. If errors persist, compare the timestamp of any "Page fetch failed" error with your Cloudflare security events to identify which rule might still be causing the block.

After updating your robots.txt file, use the "Validate Fix" button in the Page indexing report and request a recrawl in the robots.txt report. This will prompt Google to recheck your updated file, and the fetch status should change to "Fetched" instead of "Not fetched". Google generally caches robots.txt for up to 24 hours, so give it a day. These steps will help ensure that your Cloudflare settings are correctly configured to allow search engines to crawl your content.

While Google Search Console provides quick insights, automated tools can help you monitor indexing progress over time.

Use IndexMachine for Automated Indexing Tracking

To complement the manual checks, IndexMachine shows the index status of every page in your sitemaps, straight from your own Search Console, so pages stuck behind a Cloudflare block show up as a list and you can watch them recover after the fix. For a page Google turns down, one click explains the likely cause and the fix in plain words.

Once crawlers can get in again, IndexMachine gets the pages back on Google: Google's own Request Indexing in your own Search Console, on autopilot, every day, using your full daily Google quota on every site, plus unlimited IndexNow for Bing and the other IndexNow engines. Every requested page is tracked until it is on Google. Supporting Google, Bing, and even LLMs, it gives a detailed overview of how search engines are treating your site, with daily reports, a weekly email report, and alerts for issues like 404 errors, so you can address them promptly. You can pay once for lifetime access or subscribe.

How to Unblock Challenges in Cloudflare

IndexMachine data: robots.txt blocks vs firewall and server errors

We looked at the sites connected to IndexMachine: on October 31, 2026, about 1.17 million sitemap URLs on 1,513 sites, each with its latest state in Google's URL Inspection data. Aggregate counts only, with no site or URL named. Most of these sites are on free accounts, where IndexMachine has not requested anything yet, so the numbers show where sites stand before indexing is turned on.

What it tells you:

• Real robots.txt blocks are rare. Only 17 sites had a sitemap URL blocked by robots.txt. On most of them it was a single URL; one site accounted for most of the 5,241.
• Server and firewall errors hit more sites. 51 sites had sitemap URLs returning server errors and 9 had URLs answering 403 to Google. When Search Console shows a block you did not write, check the firewall and the server as well as the robots.txt file.
• Blocking crawling does not always keep a page out. On 9 sites, Google had indexed sitemap URLs that robots.txt blocked. To keep a page out of Google, let it be crawled and use noindex instead.

If the blocked pages are only part of the story, our guide to why a URL is not on Google walks through the other reasons one by one.

Conclusion: Maintaining Search Engine Access Long-Term

Addressing "Blocked by robots.txt" errors in Cloudflare requires consistent vigilance. Regular monitoring is essential to avoid misconfigurations that could harm your site's visibility. For instance, reviewing the "Pages" report in Google Search Console weekly can help you spot sudden increases in blocked URLs before they affect your rankings. If you decide to use a robots.txt file, ensure its settings allow access for both search engine and AI crawlers you want on your site.

When implementing changes, it's important to have a process in place. After updates, check Google Search Console within 24 hours to confirm no new blocks have appeared, and keep an eye on Crawl Stats for any new spikes in 403 or 5xx responses.

"Creators should be in the driver's seat." (Cloudflare)

This quote from Cloudflare highlights the importance of aligning your dashboard settings with the directives in your robots.txt file.

To avoid conflicting instructions, ensure your Cloudflare dashboard settings match your physical robots.txt file. If you're using Cloudflare's managed robots.txt or AI Crawl Control, double-check that these automated tools align with your SEO objectives. Additionally, explicitly include Disallow: /cdn-cgi/ in your robots.txt file to prevent Google from encountering crawl errors related to this Cloudflare-internal directory.

For ongoing peace of mind, tools like IndexMachine can automatically monitor your site's indexing status and alert you to issues like 404 errors. This proactive approach keeps your content visible to search engines without requiring constant manual oversight.

FAQs

How can I tell if Cloudflare, not my robots.txt, is blocking Googlebot?

To determine if Cloudflare is blocking Googlebot instead of your robots.txt file, open the security events in your Cloudflare dashboard (Security > Analytics, Events tab). Check for any instances where requests from Googlebot are being challenged or blocked, particularly on the /robots.txt path. Then compare the robots.txt file Google fetched (Search Console's robots.txt report) with the one on your server. Additionally, review your bot settings, AI bot policies, or other security features that might be restricting Googlebot's access. Tweaking these settings can help ensure Googlebot can crawl your site without issues.

Does a 403 on robots.txt block Google from my site?

No. Google's robots.txt rules say a 4xx response (except 429) is treated as if no robots.txt file existed, so Google assumes there are no crawl restrictions. A 5xx, a 429 or a timeout is different: Google stops crawling the site for the first 12 hours, then uses the last good copy of the file. A firewall that answers 403 on your pages is still a problem, but it shows up as "Blocked due to access forbidden (403)", not as a robots.txt block.

What's the safest way to allow verified bots without weakening my security?

To keep your site secure, it's smart to set up rules that allow traffic from trusted bots, like search engine crawlers, while blocking or challenging unverified traffic. For example, you can use Cloudflare's rules engine with the cf.client.bot field to specifically permit verified bots like Googlebot and Bingbot. This approach ensures that only trusted bots gain access, reducing the risk of malicious or unauthorized crawlers causing harm.

Why should I add "Disallow: /cdn-cgi/" to robots.txt?

Adding "Disallow: /cdn-cgi/" to your robots.txt file stops search engines from crawling Cloudflare's internal /cdn-cgi/ paths. Cloudflare says Google "encounters errors when crawling it", and those paths are not content you want in search. Cloudflare also notes the errors do not impact rankings, so the directive keeps your crawl reports clean rather than fixing a ranking problem.