Firewall Rule Host Rules

Hello. I’ve been having a lot of problems with bots burst crawling my website.
They will crawl 40 to 60 pages within a min.

I have firewall rules for Country block, Bad Bots, Country Challenge, WP Security. But I’m having a lot of problems in the US now with bots from bluehost, sunburn.dreamhost.com, hostgator.mx and piles of others.

I want to block them with Challenge Captcha. I’m just not sure how to do it exactly.

For example hostname: bluehost.com has many different boxes that keep doing this. Example:
box5559.bluehost.com
box5890.bluehost.com
box5214.bluehost.com

I can’t just add box5214.bluehost.com because the number always changes. I tried just bluehost.com but it won’t work. I can’t add the IP because the bot just keeps changing it. I can’t solve the issue by adding the US to Country Challenge because then I would need to whitelist all of Google’s Crawler IP’s or something like that.

There has got to be a better way. How should I add these in so they can stop crawling a pile of pages all in the same min and bringing my website down because of it? I’m sick of these bots!
box5559.bluehost.com
box5890.bluehost.com
box5214.bluehost.com
mx74.hostgator.mx
mx43.hostgator.mx
mx63.hostgator.mx
mx96.hostgator.mx
mx21.hostgator.mx
sunburn.dreamhost.com

I’m not sure if I should do
Field Hostname / Operator Contains - equals - contains - does not contain and so on / with And or Or.

I want to block all from bluehost.com / hostgator.mx / dreamhost.com and so on.
I really hope you all can help or I have no choice but to add the US to the Country Challenge and ruin my Google / Bing and all the normal crawlers because they would’nt be able to access the site.

You’d be better off challenging by ASN (AS Number), as there’s no good reason for web host IP addresses to be hitting your site.

https://www.ultratools.com/tools/asnInfo

I have no idea how to use that or what it does fully. There’s got to be away on here.

It’d be easier if you got the IP addresses, then you could plug them into the link I posted so you can add the AS Number to this firewall rule:

I’ll definitely try that but question.

Say box5559.bluehost.com IP is: 000.000.000.001 and I got the AS00001 and added it.
Now say box5890.bluehost.com IP is: 000.000.000.002 and I got a different result AS00002
Wouldn’t that just defeat the purpose? Because these IP/s and box numbers change? It would be like what I’m doing now and just block the IP’s but that’s not a solution because they still do it.

Did you get a different ASN for two Bluehost boxes?

2 Likes

It’s the same. Yay, I’ll definitely use this! Since Dec 29/2019 I’ve been getting these bots burst crawling my site like I said 30 to 60+ times. I swear it’s the same bot using different IP’s and hosts.

I have Wordfence but only starts to block after it crawled 30 times. Setting its block limit to 4 makes no difference. So I have run JA Challenge on a pile of countries. It seemed to work till it figured out the US is still open without any challenge. So I get these burst crawls 2 times a day. Some days none and some days 5. It’s random but they still do it. It’s a battle I cannot win.

This is just two of the IP’s
162.241.24.182
box5890.bluehost.com
AS46606

162.241.218.115
box5559.bluehost.com
AS46606

This is just a screenshot from WordFence as is. Some of these links don’t even exist.

1 Like

I want to say thank you. It’s blocked 12 since I added the ASN.
Time will tell though for sure.

1 Like

I’d suggest changing it to the captcha challenge since there are cloudflare scrapers that exist and can solve the js challenge.

This topic was automatically closed after 30 days. New replies are no longer allowed.