Firewall Rule ASN Block

Hello, I’ve been having bot problems since Dec 29th, and since that time I’ve been using the firewall rules to block a lot of them. Most of these bad bots are on infested servers and burst crawl my site constantly.

I’ve used country block, blocking certain pages, country captcha, and some user agents.
It’s been really effective, however, I now have problems with Microsoft Azure infested servers. The most effective way is using ASN numbers. I can block ASN: 8075 but the problem with that is search.msn.com bingbot.htm gets blocked too as it also uses ASN: 8075. Is there any way to whitelist bingbot but keep ASN:8075 in my Challenge firewall rules?

These bots keep changing IP’s and the only effective way is to block with ASN.

Try putting the Bingbot’s User Agent String in that same rule with a Not Equal (or Is Not), so that ASN is blocked unless it IS NOT the Bingbot User Agent String.

So something like this? If so I’ll need to keep checking Wordfence if it can crawl the site.

This is one of bingbots from Wordfence.
IP: 207.46.13.208 Hostname: msnbot-207-46-13-208.search.msn.com
Human/Bot: Bot
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

you should use Firewall’s known bots criteria to exclude legit bots see https://developers.cloudflare.com/firewall/known-issues-and-faq/

Bots currently detected

The table below lists known bots that Firewall Rules currently detects. When traffic comes from any of these bots, the cf.client.bot field is set to true.

ahrefs Ahrefs SEO bot
apple Applebot is the web crawler for Apple, for products like Siri and Spotlight Suggestions
archive.org Archive.org bots
baidu Baidu search engine bots
bing Bing search engine bots
feedbin Feedbin.com bots
google Google search engine bots
grapeshot Grapeshot (Oracle) SEO bots
linkedin LinkedIn bots
mail.ru Mail.ru bots
naver Naver (South Korean) search engine bots
pingdom Pingdom.com monitoring bots
pinterest Pinterest bots
seznam Seznam search engine bots
sogou Sogou search engine bots
uptimerobot Uptime Robot monitoring bots
yahoo Yahoo! search engine bots
yandex Yandex search engine bots
3 Likes

Ok, so not completly sure if it’s right. From the last setting I did, it obviously did not work. Bing called site 2 hours ago meaning my ASN was still blocking that.

I’ll continue to monitor and see if this works if I did it right :confused:

Still not crawling :frowning:

Well, I don’t know what to do. It’s not working. Have any other ideas?

You tried allowing Known Bots like @eva2000 suggested?

2 Likes

Derp. I failed to read that clearly. I just added it and it’s on. I’ll wait and see if it actually crawls now

1 Like

Well, there it is. It works. Thank you!

2 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.