Currently I am a free member, and did see that the free membership allow me to tick the “Bot Fight Mode”, which is pretty much great considering it is free, but with that being said, there is a better mode, paid mode, and I’m trying to understand the differences between these two.
Regarding the free version (“Bot Fight Mode”):
I did see that it does “javascript checks” when traffic is suspected is suspicious. Question is, is it monitoring website traffic live?. Meaning, for example, if Cloudflare notice 100 page views in 5 minutes (for example) from a specific ip, does it lunch the challenge, and if fail does it block the traffic from that source?
How does Cloudflare make sure that the traffic is indeed from a legit bot or not. Does it use reverse / forward dns checks to make sure it is no fake?.
Can I count on the free version to block any melicious crawlers that simply aim to take all data from my website?.
Following these 3 questions, what is the difference in the paid version?.
This is rate limit. Bot protection does not interact with rate limits at all.
This is one method to verify some bots such as Google crawler but that’s about it, it doesn’t work well when you want to mitigate against “malicious” bots.
No, not even the ENT version can guarantee this and no protection can guarantee it. It will block some scrapers but there is no thing such as a silver bullet for scraping protection.
The PRO version might add some extra checks but I don’t think there is any actual difference on the paid plans other than having more visibility and being able to craft firewall rules to mitigate false positives.
There is also the ML bot protection on ENT and BIZ plans which some people say works well.
If the protection does not check for rate limit, I assume a bad bot can crawl thousands of my website’s pages?.
As for the firewall rules, if i understand correctly, following the logs (which are more detailed on the paid version), I would be able to add a rule for blocking a specific ip / subnet?.
This is true but, once a bot goes past the “main” anti bot protection, you can only do so much. Rate limiting is an option that would stop greedy bots that don’t throttle themselves.
Yeah, however, IP blocking isn’t the best approach as bots are constantly changing IPs and you might end up blocking humans.