Google bots cannot reach website due to robots.txt

I have 6 sites on Cloudflare and everything was working properly when we moved the sites over. Now google is not indexing any of my sites and saying that robots.txt file is causing the problem on the hosting platform. When I download the file, nothing shows that I can see to hinder google from indexing the site. Please advise if something has changed on Cloudflare’s platform or if there is a way to narrow down what is happening to these sites and fix the problem. Thank uoi.

Check your Firewall Events on Cloudflare’s dashboard > Firewall > Overview, and use the filter to find out if there are events related to path /robots.txt. Take note of what service is blocking or challenging Google’s crawlers, if any, and adjust your settings accordingly.

1 Like

I ran a query string on the firewall for /robots.txt and robots.txt and there were no results. Any other ideas. Should I be using something besides query string?

Not finding anything on the search…any other ideas?

If you’re not finding anything, then Cloudflare is not blocking it. What specific error does Google bot shows to you?

It says your page is blocked by robots.txt. No information available for this page. It shows my site name and then nothing else except the “no information available for this page.” and You have a rule in your site’s robots.txt file* that prevents Google from reading the page (which is how we generate a description in search results). However, this doesn’t prevent Google from showing the page in Search results. Therefore if the page appears in Search results, it will appear without any description.

You have so far failed to indicate how this is related to Cloudflare. I’d suggest you head to Google Search Console community for further help.

2 Likes

Cloudflare is the hosting company. Google states the following: * If you are using a website hosting service (for example, if your site is on Wix, Joomla, or Drupal), we can’t provide exact guidance here because every site hosting provider has different ways to manage robots.txt on their service. Please search your provider’s documentation to learn how to unblock your page or site to Google. Suggested terms to search for: "robots.txt provider_name " or "unblock page to Google provider_name ". Example search: robots.txt Wix.

So the question is…how does Cloudflare as the hosting company manage robots.txt on their service? I can not find documentation to show where Cloudflare keeps this file and how to update and find this file on my site.

Google explicitly states the hosting provider is the one to contact…I have pursued that avenue. My biggest question is why all six of my sites which are hosted on Cloudflare are having this issue. 2 of them are woocommerce based. 4 of them are just html based, so the only similarity with all of them is Cloudflare. So, my question remains. How do I access this file on Cloudflare, does Cloudflare know why the robots.txt is not searchable by google when there is nothing showing in my file that google is disallowed. I just need some direction and help on how to fix this issue.

Cloudflare is not a hosting provider (unless you specifically have created a Cloudflare pages based website). If your robots.txt file is blocking Google for a site you need to edit the source robots.txt file wherever that file is hosted/exists with whatever parameters you wish it to have.

Perhaps if you post the exact error message that you are seeing and the robots.txt file in question?

2 Likes

Google bots also cannot reach my 2 website (https://dubaisearch.ae/) and (https://dubailist.ae/). I tried all methods but still my website pages are not indexed

You tried all the methods? Then there is nothing left to try.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.