I’ve been trying to troubleshoot this for a while but just can’t pin it down. Originally we had been indexed by google but it looks like sometime around mid September we started returning a 403 error and were de-indexed by google (I just got access to the cloudflare account so I’m not sure if there were any changes made around that time which threw things off). When I go to the site I am still able to load all the pages, but this isn’t ideal for SEO and third party integrations, among other things. To confirm the 403 error I looked in the Yoast WP plugin site health tool and when I performed a search at Ryte which both returned 403 errors. I am also getting “forbidden” errors when trying to hook up external integrations like Zapier. When I tried requesting a crawl on google it would give me an error message as well. When I paused cloudflare altogether it would clear the error on Yoast and google so I think it has something to do with settings somewhere in cloudflare, but I can’t figure out where.
I checked the firewall and we had a known bots rule that blocked all known bots, including googlebot, so I created an exception to that in the rule which I think helped, but it doesn’t seem to have resolved the 403 error completely. When scanning the logs I’m noticing a “definite bots” managed firewall rule but I can’t locate that specific rule to identify if that could be the problem.
I also tried purging the cache but that didn’t do anything.
Anybody have any thoughts on where the problem could be, or how I can diagnose this?