I’m having problems with bad bots at my site. They don’t obey the robots.txt file, and they click on affiliate ads. Pretty much all of the bots use a hostname with this structure: ec2-52-23-243-41.compute-1.amazonaws.com
I’ve added a WAF rule in Cloudflare for my domain, that should block these requests:
(http.host contains “compute-1.amazonaws.com”)
But for some reason, this rule doesn’t trigger. I still get these bots, but I don’t understand why. Is it not possible to block by http.host?
I think they use too many different IPs for me to block them by IP Source Address.