Hello. I’ve been having a lot of problems with bots burst crawling my website.
They will crawl 40 to 60 pages within a min.
I have firewall rules for Country block, Bad Bots, Country Challenge, WP Security. But I’m having a lot of problems in the US now with bots from bluehost, sunburn.dreamhost.com, hostgator.mx and piles of others.
I want to block them with Challenge Captcha. I’m just not sure how to do it exactly.
I can’t just add box5214.bluehost.com because the number always changes. I tried just bluehost.com but it won’t work. I can’t add the IP because the bot just keeps changing it. I can’t solve the issue by adding the US to Country Challenge because then I would need to whitelist all of Google’s Crawler IP’s or something like that.
There has got to be a better way. How should I add these in so they can stop crawling a pile of pages all in the same min and bringing my website down because of it? I’m sick of these bots!
I’m not sure if I should do
Field Hostname / Operator Contains - equals - contains - does not contain and so on / with And or Or.
I want to block all from bluehost.com / hostgator.mx / dreamhost.com and so on.
I really hope you all can help or I have no choice but to add the US to the Country Challenge and ruin my Google / Bing and all the normal crawlers because they would’nt be able to access the site.