I have a Firewall rule set to block User Agent’s that contain the strings ‘semrushbot’ and ‘petalbot’ - now this works for most of my site eg uk.domain com, usa.domain com and other subdomains and I do not see requests from those User Agents in my server logs. However for the main domain they are still getting through and I see thousands of requests in my logs.
Has anyone else seen anything like this or can anyone suggest why this may be happening?
Thanks in advance
Turns out I hadn’t added ‘www’ as an A-record (it was a CNAME for some reason and the Firewall rules weren’t applied to that).
Just posting this in case anyone else comes across similar!
A, CNAME or AAAA makes no difference to firewall rules. But if the record was 
then the traffic will not have reached Cloudflares network, and no security or performance features would be applied to the traffic.
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.