Certain URLs 403ing Google & Bing Bot

Hey folks,

I’ve been using Cloudflare for a few months but a month ago we were hit by a bit of light attack, nothing serious just a bunch of bots downloading all of our images repeatably it was the kick up the arse I needed to upgrade to Pro and enable some of the bot fighting features.

All was working well until recently when I looked at Search Console and found that we were now returning a 403 response to over 700 URL’s that Google had found,

I couldn’t see anything in my code or server that would block them so I started to check Cloudflare settings.

We have “Managed Challenge” for Definitely automated bots and “Allow” for Verified bots and no firewall rules preventing any bots.

If I go into Search Console and ask google to reinspect the URL, it does (and returns a 403) but no access request appears on the firewall.

A few last points

  • If I turn the site into developer mode, Google can get the information from the website with no errors but obviously I lose the bot protection which I needed in the first place
  • This is also affect the Bing Bot as well as Google, which is also receiving a 403 response for some URL’s
  • This doesn’t affect every page of the website, it looks like it currently affects about 10% of the website
  • On the day this occurred, I upgraded to “Pro” and enabled bot-fight (I know that Pro changes the SSL certificate and this could take some time but it’s been almost a month!)

Would appreciate any thoughts, further questions or ideas of what I can try other than turning off all bot protection!

Thanks

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.