Denial of Service Attack not being Blocked by Cloudflare

Security
#1

Ok my site has been under DOS attack continuously for the last few hours. I have tried numerous tricks including, of course, turning on “I’m Under Attack” mode. However it appears the attacker can still get through to my server no matter what I do, i.e. Cloudflare is not blocking bad requests with browser integrity checks etc. These are clearly bad requests as you can see in the Apache access logs.

These bad requests have User agents which simply read as “x”, “l”, and “.”, and also have bogus referrers like google, baidu, youtube etc:

MyAcmeSite IBQSz3zNY8rrK/3FmH5gtg - - [02/Apr/2021:20:14:53 +0000] “GET /m.index.php HTTP/1.1” 200 18886 “https://google.com” “x” {1028:22642,21625}

MyAcmeSite fTkYMwyAMMD+Xu5xV5KbHQ - - [02/Apr/2021:20:14:53 +0000] “GET /m.index.php HTTP/1.1” 200 18886 “https://baidu.com” “l” {1028:22642,25365}

MyAcmeSite KRn/U0p+1PaFKZ7b4u6RPg - - [02/Apr/2021:20:14:53 +0000] “GET /m.index.php HTTP/1.1” 200 18886 “https://youtube.com” “.” {1033:22642,25872}

So my question is, why are these requests reaching my server without first getting intercepted by cloudflare, with a simple browser integrity check among other things?

#2

Hi @goftar,

If your server using Cloudflare’s Argo Tunnel or restricting connections to only come from Cloudflare IPs? If not, then the attack may be bypassing Cloudflare and hitting your server directly, have you checked that?

3 Likes
#3

Well I assumed everything came from cloudflare because the IP address for my server has never been exposed to the public. It has always been shielded by cloudflare from Day 1.
Considering that the hacker is specifically targetting my site (because he gives notices on when he will start the next attack) suggests that somehow CF is either being bypassed or there is a hole in CF which I am not sure of yet.

Blocking non CF requests suggests that my server must be physically behind another firewall, yes? This is a remote Hosted system so I have no control over the topography.

#4

Just a clarification to my last post: I just remembered that I have confirmed that the DDoS traffic is indeed passing through CF. I see thousands for requests reported by Cloudflare during the DDos attacks.

#5

If this is the case, do you see all the requests passing the JS challenge? Is there anything common amongst them that you could use to filter them out? Alternatively you could do the opposite and captcha challenge everyone and slowly allow traffic that is not suspicious.

#6

You need to curl or chrome url bar to your maybe-not-secret origin IP, and set up a hosts file DNS entry to skip Cloudflare’s orange cloud IP address. Your server should timeout if its not a cloudflare ASN or 404 or 500s error (or safest 200 status+0 byte page body, maybe the bot has a cache) file, on every HTTP request no exceptions if not a cloudflare IP. Do NOT confuse this with CF’s CF-Connecting-IP header.

Worst case, you need to wireshark your origin server and see if someone learned your origin server’s real IP. There are enough guides on the internet describing many tricks to extract a CF origin IP from a orange clouded website that I would say you have to assume if someone really wants to they can figure out your origin server IP. Changing the server IP (a nightmare IMO) hosting provider side and CF dashboard side might be the only way to make that bot go away if it really learned your origin IP address. Basic de-cloudflare trick, your origin server also sends your weekly newsletter emails to your customers, same IP address. Busted.

1 Like
#7

Yes it looks like the majority of DDOS requests are passing the JS challenge (the evidence being that my hosted server gets overwhelmed; I am not sure how you determine whether CF is actually passing or rejecting a JS request, since the CF log entries do not appear to provide this information)

One example of a CF firewall log entry from the DDOS attackers is:
Ray ID 63ae7e3aff8542d7
Method GET
HTTP Version HTTP/1.1
Host MY_WEBSITE_DOMAIN_HERE.com
Path /
Query string Empty query string
User agent Mozilla/5.0 (Linux; ; ) AppleWebKit/ (KHTML, like Gecko) Chrome/ Mobile Safari/
IP address 47.106.91.52
ASN AS37963 CNNIC-ALIBABA-CN-NET-AP Hangzhou Alibaba Advertising Co.,Ltd.
Country China
Service Security level
Rule ID riskyiuam_bot_score
Action taken JS Challenge
Export event JSON

So it is presenting the JS challenge to the attackers’ requests (typically at a rate of over 50 per second from origins typically in East Asia, Southeast Asia etc., (identified as having a “risky bot score” from the looks of it). There is no indication of a subsequent Captcha-style challenge to the same request.
I am led to believe that the request is making its way to my server at the same rate as it is being received by CF, though again, I have no way of knowing whether or not the JS Challenges are being passed or rejected right now.

Thanks

#8

Yes that would be a clever way to reach the server with the domain name properly embedded in the HTTP request. As you probably know, my server’s IP address is “shared”, so simply doing a request to port 80 (or any other port) at that address will not be responded to. For this reason, right now I don’t think there is a direct (non-CF) access occurring at this time. I could be wrong, but due to the hosted nature of the server, I have no way of analyzing traffic unfortunately. As you indicated, there are other more tedious ways of course to make this determination and I’ll certainly pursue this route if there is definite reason to believe there is non-CF traffic reaching the site.

At the moment, I am certain that at least some of the DDOS traffic (likely 100% of it) is indeed passing through CF, though not certain whether any of it is being blocked.
Thanks

#9

So from that example, it looks like the security level is triggering a JS Challenge. From what you said about the number of requests hitting your server, I suspect it is passing the challenge.

If you can see a common pattern, such as ASN or country you could try applying a Captcha challenge (or even a block of you can narrow it down enough) to those requests using Firewall Rules.

I would recommend reading the #tutorials post I linked to earlier and also:

1 Like
#10

In Firewall → Settings, what’s the Challenge Passage time set to? I believe 30 minutes is default, but you might want to try lowering it to 5 minutes to force them to jump through that hoop again. It might slow them down.

2 Likes
#11

Is it shared hosting or not? If its shared hosting port 80/443 will always respond to any IP, it can’t timeout/ICMP “port closed” non-CF IPs. If you set up a local hosts file DNS entry for your site, and connect to the origin IP address, no cloudflare, no global DNS system, will you see your site?

The bot couldve done a reverse DNS on your origin IP address, got your official host name, then connected on port 80/443 with the correct Host: header. Since it is shared hosting you can’t wireshark it. “Zone transfer” on DNS can mass-leak in 1 shot every site hosted on that shared hosting box.

185.199.111.153 | cdn-185-199-111-153.github.com - Fastly, United States github pages, 260K domains on one IP address behind shared hosting :grin:

#12

It’s shared. Connecting directly to the IP address will not reveal my site. It might reveal some generic web page belonging to the hosting company.
So far the attacks coincide with what I see being reported in the cloudflare logs, so no reason yet to believe CF is being bypassed.
Nevertheless, per domjh’s suggestion, one option might be to have the hosting company help in blocking non-cloudflare ip addresses.

#13

That isn’t enough to prove is your site can’t be reached from outside world. putting https://123.123.213.123 or http://123.123.213.123 into an address bar 3 out of 4 times results in a server error/config error/etc. Add your domain with your origin IP to your hosts file on your PC, disable/enable network connection, and put http://yoursite.com into URL bar is the only way to prove is your traffic can be going to public internet without CF. 90% of servers on earth require the Host: header to be correct to deliver content. Cloudflare can also be sort of detected by headers if you can’t block non-CF IP addresses or your web server doesn’t give you the IP in PHP/python.

#14

IUAM only does a javascript challenge, you can set page rules for additional challenges

#15

Do you really see/realise any value in sharing your content with some countries? Worth blocking countries at firewall level?

Worth creating rules that present a challenge (not JS challenge) if threat score is equal/greater than 2?

The suggestions above still apply too - even if you can’t block non-Cloudflare traffic at ISP/host level you could create rules to reject requests that are not from CF at domain level, surely? Direct traffic could still hit your server but would not spend processing time.

#16

The attackers I’m dealing with have thought of that. The DDOS flooding comes from multiple countries including U.S., often simultaneously. Other times I see them acting in series: i.e. I block one attacker and immediately a different IP address from a different country resumes the high-rate attack.

The threat score needs more explanation. Like who would I be challenging/blocking that has a threat score of 1 or 2?

#17

More information on using Threat Score: https://developers.cloudflare.com/firewall/known-issues-and-faq#how-can-i-use-the-threat-score-effectively

It is similar to Malicious IPs | By Last Bad Event | Project Honey Pot as it assigns a score to inbound connections. Anything above with a score above 2 could potentially be dangerous but your experience may be different. I have a rule to block everything above 2 for my domains.

It won’t stop everything but might be another thing to use - check your firewall stats after a while to see if it works for you.

1 Like
#18

After some more research I realized the reason some of my FW rules are being ignored are cases where I have set some ASN’s to “Allow” under the FW Access rules. The reason I do this is to temporarily Block some of these bad-acting ASN’s when there are periods of intense abuse.
So does anyone know whether I can set up FW rules in a way to override these "Allow"ed ASNs? I suppose the alternative is to delete the ASN’s from the FW Access Rules List which would require me to manually add them again to the list later when necessary…

#19

Set the rule to block. Turn the rule off in stead of changing from Block to Allow when you no longer want to block it.

1 Like
#20

@goftar If you have not already (which I think you have) activate IUAM (I’m Under Attack Mode) so it checks EVERYONE who visits your site, also what’s the link. And as mentioned, try making firewall rules to block or challenge those countries that are attacking your site. Also rate limiting maybe helpful!

1 Like