Genuine Googlebots are being being periodically 403’d by the edge servers - it’s not all traffic, just intermittent and can be easily seen by filtering on analytics. I have raised this issue a week ago, but since Cloudflares move to Salesforce, their support seems to have disappared. Gone are the days of reply in 24 hours with that upgrade.
What steps have you taken to resolve the issue?
It can clearly be seen by running the filters on analytics dash.
Was the site working with SSL prior to adding it to Cloudflare?
It appears that you’ve enabled the Definitely automated under Bot Management. You can be reviewed in Security Events section, where requests are flagged and blocked by managed rules. If you’d like to modify this behavior, you can consider adjusting the action to Managed Challenge or Allow for this feature and then monitor the impact on your traffic.
I definitely do have that enabled - however I didn’t enable it to allow it to block genuine Google crawlers.
Are you 100% sure you’re answering this question correctly as I should think no one is enabling your bot management assuming it’ll be blocking genuine Google crawlers which you can easily identify? That surely renders the whole of your bot management useless doesn’t it?
Why would anyone want to be blocking a public facing site from Google crawlers?
I have a WAF rule to allow known bots which surely should negate all this anyway?
Please can you categorically confirm that your bot management will block Google crawlers when you block ‘definitely automated’ bots even though you have a WAF rule to allow known bots.
Also this is the description of definitely autoated which doesn’t fit Google - “Definitely automated traffic typically consists of bad bots. Select an action for this traffic.”
And the WAF rule should skip “All Super Bot Fight Mode Rules”
Are you sure your detection of genuine Google crawlers just isn’t working as it should?
I’ve also confirmed with Google themselves that the issues we’re seeing with one of their products is caused by 403 errors which led me to find this in Cloudflare.
That link wouldn’t be accessible to me as I don’t work for Cloudflare nor do I manage that domain. And the source is Bot Management? Sorry you’ve not provided details on the WAF event so……
If that’s the the case the support ticket would be the way to go, the community doesn’t have the ability to change anything