After seeing so many posts here on “How do I stop an attack” I’ve noticed some admins are calling “attack” what could simply be aggressive scans (either bad behaviour search bots or malicious scanners looking for vulnerabilities). I decided to post a new topic with an example of the firewall rules we use against such scans.
Some are self-explanatory but here we go:
- Allow PayPal IPN
- Allow a known, good IP address for testing
- Block requests on ports that aren’t 80 or 443
- Block TOR, and countries currently sanctioned by New Zealand or the UN or others in our block list
- Allow robots.txt for known bots (important because later we block some known bots that we don’t want anyway and they should access robots.txt to “learn” they are disallowed)
- Allow HSTS checks
- Automated traffic we don’t like (the list exists in another post but now using regex instead)
- Block anything that is not HEAD, GET, POST
- User agents that contain bot names but are obviously fake (wrong IP, country, etc)
- File used for traffic management but only for cloudflare
- Block sitemaps for non known-bots
- Block access to things we don’t have - wp-admin, phpadmin, umbraco folders, etc
- Block access to some other paths we don’t have
- Block Fake search requests
- Block additional SQL injection statements that Cloudflare is not blocking
- Block HTTP 1.0
- Block registration from some countries where a huge number of spammers come from
- Block threat score above 1
- Block Internet Explorer 9 and below plus a couple of metasploit UA
- Block attempt to use our image uploader with invalid parameter or request method
- Block access to a specific path unless with some parameter and request method
- We have some XHR calls with some parameters - block anything missing those
- Sometimes known bots try to follow a path they shouldn’t - block those
- Sometimes known bots try to load our search page - block those
- (inactive) In case of an attack we can block everyone from posting or logging in
- (inactive) In case of an attack we can block everyone from accessing the site if not in AU/NZ
- Allow known bots - this will allow most good bots that reached this point unless they were blocked before (which are the ones we don’t want)
- Block cloud services that are not known bots or not specific user agents (monitoring services)
- Challenge if request previously failed reCAPTCHA (logged in users)
- Challenge if request previously failed reCAPTCHA (all users)
Some rules are very specific to our use case. For example path or reCAPTCHA monitoring.
The regex we use to block bots we don’t want is a version of the posts here
In addition to those rules, we block a few ASN and IP addresses. The last 72 hours these rules together blocked 600k bad requests.
What this shows too is that if we relied only on the basic WAF without any custom rules our servers would be swamped with bad traffic.
It also shows that relying on user agents, IP or ASN alone won’t stop those scans from happening. You need an integrated approach and need to understand your traffic.
Some rules show low numbers on these screenshots but they exist because they are triggered at some point.
We constantly monitor our logs using Datadog and have some alarms set based on number of requests - anything that goes over the threshold is analysed and a rule is updated or a new rule put in place.
I would love to replace Google reCAPTCHA with Cloudflare Bot Score being available in headers or firewall expressions - alas this is only available to enterprise clients and the Super Bot Fight Mode is a poor attempt at making something for the mid-tier online properties. Because SBFM cause so much problems I have it OFF for all use cases.
For these rules to work you should always make sure your server only accepts inbound connections from Cloudflare IPs listed here IP Ranges | Cloudflare - even better if you could block anything but these IPs at a firewall before it hits your server.
We never had to use rules 25 and 26. Cloudflare would kick in beforehand or some other rule would prevent the need for it. For example, this was a short attack prevented with these rules: