Effectiveness of Bot Fight Mode

A few years ago, I found myself under attack from a scraper bot that changed IPs each time I blocked it. I used the API to block this bot which I was able to detect by a signature, and within seconds it would change IP. Not sure how they did that, but I saw that their requests came from hosting companies and I decided to use the API to ban any ASNs used by the bot, as my site is aimed at real users not bots. My custom 403 page invites blocked human users to get in touch, which occasionally someone does (nearly always VPN users) and I have never looked back or seriously felt I was missing legitimate traffic.

Blocking hundreds of ASNs was the nuclear option and I went for it. Now, I see there is Bot Fight Mode.

My question is, given that my Firewall security level was already set to Medium and plenty of bots were getting through, how different will Bot Fight Mode really be? And, if I decide to try it, can I temporarily disable my hundreds of ASN rules without losing them?

Any comments are welcome! Thanks!

Try setting a Captcha challenge instead of blocking the traffic. This will most likely stop the bots and and allow the users on a VPN to solve the Captcha and enter your site. I do not believe bots can solve the captchas anymore, so it won’t hurt to try this.

Thanks, @smalldoink, I will do this for some of the ASNs, and this is, or would be, a reasonable solution except that the only bots I really care about are those that are intent on copying my site. The one I described above made a big effort to get at our content and I believe a keen player would find a way to solve a captcha - perhaps by browsing our site using their server as a proxy?

I just need to scare off potential troublemakers without harming real human users, so I may prune my list of ASNs and keep only the big cloud hosts - Digital Ocean, OVH, etc. and set a Captcha challenge for some others.

Usually, I find out about the copying by searching Google for watermarks in our content. Then I can research and prevent further leakage, as our content ages quickly.

Any more suggestions are welcome, thanks in advance.

I suggest pushing a captcha challenge for any and all cloud servers you find. Some people use residential proxies (Which are proxies from home connections), but i doubt any of the scrapers are using these types of proxies. It will be a slight inconvenience for users browsing with a VPN, but it will help protect against the bots.

I see no real world change after bot fight mode. Thing going as usual before or after bot fight mode.

This topic was automatically closed after 30 days. New replies are no longer allowed.