FW rule blocking Microsoft ASN without blocking Bing

[irene]

Hi I want to block Microsoft and Google without blocking Bing or GoogleBot.

I did the the following:

However its not working with Microsoft:

What am I doing wrong?
Thanks

[fritex]

You might want to switch the toggle button from green to gray.
Therefore, make sure the method is “block”.

However, I might not be sure how Cloudflare checks this, by user-agent or IP range

So, maybe like writing your own Firewall Rule … for example:

(http.request.uri.path contains "sitemap" and ip.geoip.asnum ne 15169 and not http.user_agent contains "Googlebot") or (http.request.uri.path contains "sitemap" and ip.geoip.asnum ne 8075 and not http.user_agent contains "Bingbot") or (http.request.uri.path contains "robots.txt" and ip.geoip.asnum ne 15169 and not http.user_agent contains "Googlebot") or (http.request.uri.path contains "robots.txt" and ip.geoip.asnum ne 8075 and not http.user_agent contains "Bingbot")

With the method “BLOCK”.

Using above, would be you’re allowing regular and valid Googlebot and Bingbot to visit your robots.txt file and crawl index your sitemap file(s) if any existing, while blocking even yourself out of trying to access them.

Haven’t tried it yet, but just thinking

You’d

[paul32]

If you want to block Msoft I’d also block AS8068 through AS8075
We block all Msoft traffic as 99.99% of it is attack attempts and Msoft ASN’s are responsible for about 80% of attacks on our sites
40.122.76.165 resolves to ns2-201.azure-dns.net not a name ending in search.msn.com so isn’t a bingbot - See Bing Webmaster Tools & Network Tools: DNS,IP,Email

[irene]

You might want to switch the toggle button from green to gray.

That’s the mistake! Thanks, that was really useful!

[irene]

If you want to block Msoft I’d also block AS8068 through AS8075

Thanks!

40.122.76.165 resolves to ns2-201.azure-dns.net not a name ending in search.msn.com so isn’t a bingbot - See Bing Webmaster Tools & Network Tools: DNS,IP,Email

I know, that’s why I tried to block it.
I didn’t knew that the known bots button has to be gray for this rule, as fritex pointed

[irene]

However I still see IPs that belongs to blocked ASN coming to the site and causing 404 errors.

LOG

IP

FW rule

[paul32]

Which rule comes first? Your “block ASN’s” rule needs to be above the “Allow BingBot & Google Rule”

I find it easier to just add the ASN’s to the Security/WAF, Tools form - also gives you the advantage that you can ban the usual suspects from every website in your account in one step, particularly as over time you will probably add a lot of ASN’s to be blocked, and you save having a huge rule with lots of ASN’s listed