We’ve been experiencing indexing issues with search engines which has lead us to discover that bots are being blocked by CF; Bing has confirmed they cannot access.
When the site is visited as a bot, an error is displayed indicating “…you have been blocked”.
There is a RayID, but it, and the block itself, never show up in the firewall log.
This is a basic account with the Cloudways CF integration enabled.
I have no firewall rules, no IP access rules, no User Agent blocking rules and Bot fight mode is off.
Looking at the firewall logs, there’s nothing with my IP or that RayID.
I have tested WordPress in troubleshooting mode to make sure it’s not a plugin.
I have also tested disabling Bot protection in Cloudways (whish also shows nothing).
Bing you say? What is your TLS on? 1.2 minimum? If you have it set on 1.3, Bing will not be able to enter your site since it does not support it yet, only Google Yandex do.
Had this occur to me on some other sites, and seen others here affected by it.
Its a longshot, thought I would mention it. TLS minimum settings are in SSL/TLS > Edge Certificates
Thank you for the reply. Since you say: there’s no information in your CF dash on RayID that’s shown when the site gets blocked, its more likely not bots, could be some malware infection. The image you’ve posted here could be fake & not from CF. That’s what I’m guessing when you make that the above statement.
If not, would you check again? There must be some logs. Here’s one guide:
There is a RayID that is shown when blocked, but no corresponding log to that ID in the account to show why it was blocked.
Following up on the malware idea, I tested just basic files like a test.txt and robots.txt, they also get blocked, as long as my user agent is a bot (tested with Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)).
I placed a file called test.txt with simply “It worked” in it.
If I browse to the file in a normal browser, it loads just fine.
When I act like a search bot (google), I am blocked.
The RayID generated is not found under the account logs.
Just to share, this turned out to be related the the way Cloudways and CF work together.
Disabling the integration between the 2 solved the issue with no more 403’s.
