Today one of our servers was registering a load of 50 as observed by ssh: top
Investigation lead us to the server logs which show the server being bombarded by IP addresses like 172.70..
Every IP that we’ve done a geo-ip for has shown the same thing:
location: Singapore, Singapore, Asia
ISP: Cloudflare
We did an iptables firewall to block the entire 172.70.. subnet and within a few minutes the server load went back to normal.
A few hours later I was live monitoring the server and it happened again! Server load spiked from about 2-3 to 50 and the server became almost non-responsive. We had to firewall the subnet again for about 10 minutes.
Does anyone know what’s going on? We have lots of servers and sites with CF integrated and we’ve never seen this before.
We’re just looking directly at the site’s access log. It’s the only site on the server.
It shows tons of different IPs with GET requests, and they all start with 172.70.143.x or 172.70.142.x
They’re looking for files like /web.rar /web.zip /webroot.rar /webroot.tar.gz
Many many every second.
Yes - I would expect normally those to be users who access the site and so CF directs them to the server and then the server sends the page/file back. But there seems to be something awry here.
Here’s a snapshot of the log.
It looks like a brute force of some kind, but it’s either from CF or coming straight through CF.
We have attack mode on. It didn’t make a difference.
Look to me someone might be using Cloudflare WARP to “hide behind” running a scan/try, or rather you did not implement returning original visitor IP at tyour origin as follows at the below article?:
All the requests are for non-existent files, resulting initially in 404 errors generated by wordpress from what we gather, but then when the server gets totally overwhelmed then they end up at 500 errors.
All DNS records are proxied except for a couple like the mail server or sendgrid which can’t be.
I’m reviewing your first link at the moment.
I’m not sure it’ll work on our systems yet as it’s CentOS, but perhaps.
And your 2nd link - there are possibilities there. We’re assessing.
Will look at your 3rd shortly.
Useful trick is to block some known ASNs or user-agents, or even countries?
Regarding WordPress, kindly may I suggest to have a look at my below post as far as it contains “all-in-one” useful links (which explain and contain screenshots “how and why” to setup) in terms of WordPress security using Cloudflare - at least I believe are a good start to implement (or to think about)
Nevertheless, re-check and make sure it’s all SSL too (just in case):
I would also recommend using a Cloudflare Pro plan as it has got Managed WAF Rules (for PHP, WordPress, OWASP, etc.) which you can enable in a second with a single click.
including more Firewall and Page Rules to configure, better image optimization and delivery like Polish and Mirage, etc.
A good note is to keep an eye on Firewall tab → Firewall Events log. Therefore, you can easily recognize and block any of the suspicious requests for further cases.
Correct - all is SSL.
Nice guide. There’s a number of elements in there we do standard as well. We’ll have to look at a few other elements there that are new, though.
We have special iptable country blocks already on our systems, too.
But in this case it all appears from CF, so it’s going straight through everything. We’re still looking at the reverse IP modifier, but when attacks come in we can’t even run commands.
The top server load was 100 a few minutes ago. I didn’t even know it could go that high. SSH and everything froze.
I’m quite sure that’s all proxied traffic and your server is not configured to restore original Visitor IP addresses, as suggested by @fritex. So whatever you see in your server log isn’t telling us much other than it’s possible the attack is coming through Asia.
Correct.
ngx_http_realip_module is now enabled so real IPs are being seen in apache and ngnx logs.
A new CF firewall is setup to block all http 1.0 requests, although oddly that didn’t seem to change much.
A new CF firewall is only allowing US connections at the moment. Will open it up more shortly.
It’s quite possible that your log is showing Cloudflare’s 1.0 connection to the origin when the attackers may be using 2.0 to the proxies. Though that doesn’t quite make sense to me as I look at my logs full of HTTP/2 requests with some HTTP/1.1
In my defense, I’m using Cloudflare Tunnel, so I’m not sure where the logs are getting each little bit of that data.
under the DNS tab of Cloudflare dashboard for your domain name?
