Using custom rules in WAF to block specific user agents. They still go through. I select “User Agent” > contains > string (i.e. Keydrop, zgrab, etc.), or I select URI Path > contains > env, yet I still see env causing 404s on the website. I also tried by blocking IP ranges, they still come through as I see them creating 404s. Why is this not working? It should be as simple as it looks when setting it up in WAF Custom Rules.
Thanks. Before I continue, I have to iron an important finding, and that is that some of these requests are direct to IP. As far as I know direct IP calls will bypass Cloudflare, right?
In plugin I use in WP, the 404 logs do not show the domain name/IP, but just what comes after. But, the access logs in the same plugin show it.
I have to figure the best way of blocking requests that go directly to IP.
Yes, you should use your firewall to allow access to your webserver’s ports only from Cloudflare IP addresses so you know requests can only come via Cloudflare.
Thank you. I guess these addresses may change at times, so we need a firewall that uses Cloudflare API.
Added note: After more reading I realized all the IPs I see are not original ones, so blocking by IP is pointless, they are all from CF unless it was direct access via my web server’s IP.
But I am still a bit puzzled as I also blocked some IP ranges, and it turned that some of them are used by Googlebot, and they are coming through.
It would be those first two ranges in particular: 172.70.0.0/16 and 172.71.0.0/16
Googlebot keeps coming while using IP addresses from these two ranges. Not that I want to block it, this was by accident as I had suspicious requests coming from these ranges. The site was hacked last month. A hacker was able to serve thousands of pages for various products, it was in Japanese. Some sort of scam I guess. I removed all what I could, it seems clean now, and that is why I am now trying to do everything possible so they don’t come back. WordPress is always a target. Google is coming back for these pages that do not exist anymore.
I will eventually delete these ranges from my Custom Rules, but what’s the point if they are coming through anyway. I am puzzled for this part.
The screenshot shows you using them in the Cloudflare WAF. You need to use them in your server firewall instead to protect from requests going direct to your IP address.
If you are getting confused about seeing Cloudflare IP addresses in your server’s log, those are requests that are passing through Cloudflare. You need to restore original visitor IPs so you see the real client IP addresses…
