I’d like to avoid web scrappers from stealing contents from my site. Before being working with Cloudflare, I had a PHP script which checked if an IP address was a known search bot. If not, it checked if such IP address crawled my webpages with a high speed rate (e.g. 100 pages per minute). If so, it was banned.
After starting to work with Cloudflare, I stopped that PHP script since --obviously-- it was only seeing CloudFlare IP addresses, and I cannot distinguish between a legit Googlebot visit and a malicious scrapping script.
Now I wonder about the best way to protect my website against these scrapping visits. I’ve seen that I can configure the Firewall options (Pro Plan) with a ‘Security Level’ and a ‘Bot Fight Mode’, and wondered if there are experiences about the best options.
Thank you very much.