Cloudflare Causing 403 Error

I’ve been trying to troubleshoot this for a while but just can’t pin it down. Originally we had been indexed by google but it looks like sometime around mid September we started returning a 403 error and were de-indexed by google (I just got access to the cloudflare account so I’m not sure if there were any changes made around that time which threw things off). When I go to the site I am still able to load all the pages, but this isn’t ideal for SEO and third party integrations, among other things. To confirm the 403 error I looked in the Yoast WP plugin site health tool and when I performed a search at Ryte which both returned 403 errors. I am also getting “forbidden” errors when trying to hook up external integrations like Zapier. When I tried requesting a crawl on google it would give me an error message as well. When I paused cloudflare altogether it would clear the error on Yoast and google so I think it has something to do with settings somewhere in cloudflare, but I can’t figure out where.

I checked the firewall and we had a known bots rule that blocked all known bots, including googlebot, so I created an exception to that in the rule which I think helped, but it doesn’t seem to have resolved the 403 error completely. When scanning the logs I’m noticing a “definite bots” managed firewall rule but I can’t locate that specific rule to identify if that could be the problem.

I also tried purging the cache but that didn’t do anything.

Anybody have any thoughts on where the problem could be, or how I can diagnose this?

Make sure you don’t have any Bot Fight Modes enabled in the Firewall Section. It has its own “Bots” subsection.

1 Like

Ahhh, yes, that is where the “definite bots” rule is housed. When I turn that off it does resolve the 403 error. Thanks!

It does mention, however, that traffic filtered out with the definite bots setting is mostly bad bots. We also have a known bots custom firewall rule that is activated with an exception to allow googlebot access. Does anyone know how the Known Bots traffic differs from the Definitely Automated traffic? Is it just new (unknown) bot traffic that would be able to get through? Is that a significant amount?

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.