Reaching robots.txt failing in both Google Insights and Bing Webmaster Tools

Hi Cloudflare community,

I am experience an odd issue, with robots.txt that periodically can’t be accessed by Google Search Console, Google Speed Insights and Bing Webmaster Tools. After I turned on Cloudflare DNS this problem started.

Any idea how this can be fixed? Clearly there is no indication why it is happening, but it not happening all the time. I looked at some of the previous threads in the community which were less than helpful.

Your input is highly appreciate it.


Hello. I noticed the issue on Google Search Console on my pet project only when got more than 100 pages that were not indexed (because of some problems). I tried to reindex pages via GSC and got a few strange error reasons. Not accessible sitemap and robots
After a while, I found that I had to disable some security settings for the /robots. To prove it - first of all I tried to disable security for the /robots

After that I re-scanned problematic pages on GSC and got all green. 2h before it was red. So I assume that is a reason.

Next step - is make smart security settings for robots url