Google bot not detected correctly

[oferlaor]

What is the name of the domain?

htmag.co.il

What is the error number?

403

What is the issue you’re encountering

google bot getting a 403 error due to bot fight mode

What steps have you taken to resolve the issue?

I had to ALLOW “definitely automated” bots

Was the site working with SSL prior to adding it to Cloudflare?

Yes

What is the current SSL/TLS setting?

Full

What are the steps to reproduce the issue?

I keep getting errors from Google crawler complaining it can’t crawl my site due to 403 errors.

I ran a trace for this URL: מסך גיימינג Asus ROG Swift PG259QN 24.5'' 360Hz כ1985 ש"ח - מגזין HT, using User-Agent: Googlebot/2.1 (+http://www.google.com/bot.html). The trace came back with 403

In Bot flght mode under “Definitely Automated” I had Managed challenge before.

After switching to “Allow”, it now works. Validating now with Google.

Screenshot of the error

[fritex]

@oferlaor

google bot getting a 403 error due to bot fight mode

May I ask for any more details to share here wit us of the blocked request Googlebot from the Security tab → Events?

[oferlaor]

it’s very hard to find on the events because I don’t have much info about the googlebot and what it’s seeing or when exactly it pinged a particular URL, I’ll turn it back to “block bad bots” and see if I can reproduce in events.

[freitasm]

You won’t be able to “reproduce” because the Google Bot is verified by both user-agent string and IP addresses (https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot).

Your test will fail because the IP address is not in that list, even if you use the correct user agent.

You can find if the Googlebot is being blocked by using the Security Analytics page, filtering by Mitigation Service: