Site under DDOS attack, “I'm under attack” mode not working

My site is under denial of service attack again, and placing it under “attacked” mode isn’t helping at all.

The attacks involve basic main page loading from hundreds of sites around the world including the US which slows down the site and none of the requests appear to be getting blocked by cloudflare even under attack mode.

I have added firewall rules which trigger on some of the attacks. but that doesn’t cover all of them and I’m posting this message to ask why cloud flare firewall is responding with “managed challenge” in spite of the fact that I have selected captcha challenge to make it more robust?

posting under general because there doesn’t seem to be any better category for this question.

I would suggest checking this guide out.

If that doesn’t work, we can suggest some rules, but I’d ask for the following:

  1. Your current plan
  2. The current firewall rules that you deployed
  3. Photos of the dashboard, showing the firewall, analytics. The more information, the better.

Backdoored websites are attacking your site? Weirdly, they managed to bypass the JS Challenge; however, it’s doable.

7 Likes

I have several rules which are set up to present a captcha challenges to TOR users (because, currently, Tor nodes are where the vast majority of the attacks are coming from) as well as presenting captcha challenges to nodes in countries that have a high Attack:Legitimate ratio.
These rules are triggering hundreds of thousands of times, but they are doing the “Managed Challenge” routine (which I suspect is JS challenges). This is one of the issues that I don’t get; Why is cloudflare firewall doing “managed” if I have set up the rules for Captcha challenges?
I am on the free plan.

The attacks only concentrate on the root /

These triggers are on my Country-based firewall rule, which is supposed to present captcha challenges, but apparently it is not:

Right now TOR nodes are the major source, but the attacks are dynamic and these can change. They used to come from Cambodia, and China in the last round. The attacks are trigged by a specific person that appears to have software to specify the name of the web site for all of these nodes to attack

I have currently set security to “Medium” since "Under Attack mode and “High” do not seem to make any difference in stopping these attacks. I have currently also set up the challenge passage intervals to 1 year to minimize inconvenience, and setting it to shorter intervals does not appear to to make a difference (at least not in an immediate sense):

Also here is an excerpt from the server logs during the attack where you can see most of the user agents are Fedora and Ubuntu based. My firewall rules already were set to present captchas for these UAs but cloudflare is doing “Managed” challenges as mentioned earlier, which may not be captchas. Either way they are sliding through the firewall by the thousands:

Summary

example.com 5jeeI/qGu/j26W/DoaZpNA - - [28/Dec/2021:19:52:45 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36” {1453:5256,7593}
example.com UCbmQQCleqPmv1wnbIUKaQ - - [28/Dec/2021:19:52:45 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0” {1431:5256,9343}
example.com xFWxpYck80Yap84FbQ3HqA - - [28/Dec/2021:19:52:45 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36” {1453:5256,9596}
example.com xGxLQJzb+b9swstiD06nAQ - - [28/Dec/2021:19:52:46 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0” {1427:5256,9365}
example.com RJE6vx38xvSQlnafZFPm1Q - - [28/Dec/2021:19:52:45 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0” {1448:5256,1009684}
example.com i0kxQYZXWBtFUH5f8uBs5Q - - [28/Dec/2021:19:52:46 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15” {1464:5256,7981}
example.com CWUHpE09r/vWjrBWVz9EHA - - [28/Dec/2021:19:52:46 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0” {1414:5256,12272}
example.com E+RbitkaN5FOYwghD+mcwQ - - [28/Dec/2021:19:52:46 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (X11; Linux i686; rv:91.0) Gecko/20100101 Firefox/91.0” {1436:5256,7854}
example.com V6JBCjAyZbRnHh11G1azhg - - [28/Dec/2021:19:52:47 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/600.5.17 (KHTML, like Gecko) Version/8.0.5 Safari/600.5.17” {1480:5256,8886}
example.com 4huwxNjPLqZ4MaIZL2fPkw - - [28/Dec/2021:19:52:47 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 12.0; rv:91.0) Gecko/20100101 Firefox/91.0” {1430:5256,8924}
example.com i5SgbHQkcetYXaoHWwaq4Q - - [28/Dec/2021:19:52:47 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 12_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 Edg/96.0.1054.34” {1504:5256,8519}
example.com G07q3DFD9YgKbTyUuCJ/4g - - [28/Dec/2021:19:52:48 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 12_0_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15” {1469:5256,8743}
example.com 3ENRSlxLUdOiBMtIp3w/Uw - - [28/Dec/2021:19:52:48 +0000] “GET / HTTP/1.1” 302 - “https://www.example.com/” “Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0” {1427:5256,9527}
example.com LZpDZ8yv4C97AcBRIn+V5Q - - [28/Dec/2021:

My IP address has never been exposed to the public and I have no reason to believe these are back door attacks (but I suspect by “backdoored” you might be referring to something else).

Thanks!

If the managed challenge is showing up, I suspect that it’s because Cloudflare is delivering a challenge before your firewall rules are evaluated.
The managed challenge is “dynamic” I believe it can be both JS and CAPTCHA.

While the PRO package does not provide many features when it comes to automatic DDoS mitigation, it’s good at giving a good overview and ideas to pattern match the attacks. It’s a bit complicated to dig into the actual behavior without the analytics that the dashboard provides.
An alternative could be a grafana dashboard that shows the same as the PRO/Business plan.

I agree that this seems to be a design mistake, it is a shame that users aren’t able to know if an attack is solving Captchas or JS challenges.

Regarding the managed challenge “formulating” a bypass for your firewall rules, I’m not sure if that could be the case… perhaps somebody else can confirm this for us, @MVP. I believe that there was a discussion about this 1-2 weeks ago.

Just to confirm, are you confident that those logs are part of the attack? If so, an aggressive rule could aim towards HTTP 1.1 and the other patterns that they show (country, threat score, etc).

3 Likes

I am sorry to hear this.

How this went?

I might suggest to you to try out a Pro plan and Managed WAF Rules which can be enabled with a single click just in case for any other type of attacking requests.

I see “1 year” from your screenshot. Why so long allowing the TOR users?

Better to block all TOR (country T1) of them with a Firewall Rule in your case.

Furthermore, Bot Fight Mode might be have something to do about it - despite the “I am under an attack” option.

Meaning, all of the request to the URL / are either “success” and bypassing Cloudflare’s system somehow while the hostname’s are proxied :orange:, hitting directly your origin host / server?

Redirect from what to what?

Could you share this?

2 Likes

Hi,

Managed Challenge may show either a Captcha or a JS Challenge, depending on Cloudflare’s evaluation of the request (and the requester). Please see:

Managed Challenge help reduce the lifetimes of human time spent solving Captchas across the Internet. Depending on the characteristics of a request, Cloudflare will perform the following actions:

  • Show a non-interactive challenge page (similar to the current JS Challenge)
  • Show a Captcha

Also:

For domains on Free plan, any Firewall Rules set to Challenge (Captcha) have become intelligent Managed Challenge . As a free customer, you cannot opt out of Managed Challenge .

Source: https://support.cloudflare.com/hc/en-us/articles/200170136-Understanding-Cloudflare-Captchas-Managed-Challenge-and-Challenge-Passage

What you can do is use the Firewalls > Tools > IP Access Rules to set protections based on User Agent (limited to 10 on the Free Plan), IP Address, or ASN. There you can chose Challenge instead of Managed Challenge if you feel like it.

Another thought: Are you using a Page Rule setting a Security Level for the whole domain, or the home page? Security Level set at a Page Rule will prevail in relation to the general Security Level setting at the Dashboard. So if you want a Security Level of High or I’m Under Attack, you need to make sure no page rule is overruling this setting.

As suggested by @fritex, you should reduce the time it takes for a new Challenge to be served, and perhaps block/challenge Tor visitors.

From the log extract posted, it seems most are being redirected to add the trailing slash. You could use a Transform Rule to add the slash before it hits your origin.

3 Likes

I think Managed Challenge means not solved as the FirewallMatchesActions at https://developers.cloudflare.com/logs/reference/log-fields/zone/http_requests lists the following options

  • unknown
  • allow
  • block
  • challenge
  • jschallenge
  • log
  • connectionClose
  • challengeSolved
  • challengeFailed
  • challengeBypassed
  • jschallengeSolved
  • jschallengeFailed
  • jschallengeBypassed
  • bypass
  • managedChallenge
  • managedChallengeSkipped
  • managedChallengeNonInteractiveSolved
  • managedChallengeInteractiveSolved
  • managedChallengeBypassed

If a request did solve a Managed Challenge, the Action taken listed in Firewall would be listed as either managedChallengeNonInteractiveSolved or managedChallengeInteractiveSolved instead of Managed Challenge I believe not that I’ve seen those 2 Action taken labels as Cloudflare’s probably pretty accurate in triggering the Managed Challenge to bad requests so rarely would have solved Managed Challenges.

FYI, here’s an example tally of my Cloudflare Firewall Events log via Enterprise logpush

find /home/cfcmm-fw-logs/ -type f -name "*.log.gz" | parallel --pipe -N 3 --roundrobin -j 3 parallel -j 3 'pzcat -p1 -f {}' | jq -cn --stream 'fromstream(0|truncate_stream(inputs)) | "action:\(.Action)"' | sort | uniq -c | sort -rn
 468858 "action:allow"
 222816 "action:log"
  42087 "action:block"
    171 "action:challenge"
    100 "action:managedchallenge"
     41 "action:managedchallengebypassed"
     34 "action:jschallengebypassed"
      4 "action:jschallenge"
      3 "action:challengesolved"
      1 "action:jschallengesolved"
1 Like

Ahh that explains it. I wish they were more clear about that because the settings lead the user to believe that they really are invoking a (more robust) Captcha challenge when they really aren’t. This must be a relatively new development. I am fairly sure that, previously, a Captcha challenge meant a Captcha challenge even for free accounts. Either way that’s no longer the case…point taken.

Another thought: Are you using a Page Rule setting a Security Level for the whole domain, or the home page? Security Level set at a Page Rule will prevail in relation to the general Security Level setting at the Dashboard. So if you want a Security Level of High or I’m Under Attack, you need to make sure no page rule is overruling this setting.

To expand on this thought, I suspected one of my own firewall rules might be overriding cloudflare’s DDOS rules, but not sure yet. I’ll continue to investigate.
Looking at this diagram, it is not clear where CF’s DDOS rules kick in with respect to my firewall rules, and whether or not my own rules can potentially override CF’s managed rules in a bad way which lets the attacker slip through.


Does anyone know if this is the full picture, and whether or not DDOS protection is part of this chain, and if so, where it is in the chain? The WAF managed rules at the end perhaps?

DDoS protection should be triggered in front of anything, but yes somehow the diagram does not show that.

1 Like

Yes, and yes. Managed Challenge is a recent thing, and a Challenge necessarily meant a Captcha before. I agree with you that CF should make it more clear for Free Plan users that when you are setting a Challenge, it’s actually a Managed Challenge.

2 Likes

This….

2 Likes

I forgot to bring that up earlier.

The main resilience of the CAPTCHA challenge is the cost that it has for the attackers to solve it. If you have a 1-year pass, then the attack becomes ridiculously cheap.

If the attackers still bypass the CAPTCHA, the only options available are rate limit and blocking the attack in the most granular way possible.

2 Likes

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.