Mitigating an HTTP DDoS Attack manually with Cloudflare

[jnperamo]

This guide is for those users of Cloudflare who experience medium-high level complexity DDoS attacks.

Continue reading if you want to accomplish the following:

1. Becoming more familiar with the Cloudflare Dashboard and crafting custom firewall rules.
2. Understanding the standard behavior of DDoS attacks and deploying effective firewall rules.
3. Realizing how powerful and valuable Cloudflare Firewall Rules are.

I initially thought of making a more complex guide (I will). However, I realized that not everybody gets beyond-normal complex attacks.

This guide is for those that experience slightly more complicated attacks and has limited tech stack experience.

In future versions, we will deal with real-time monitoring while using Cloudflare (Grafana) and how to configure a machine properly so that the logs displayed are ready to craft firewall rules.

Step 1. Receive a DDoS attack.

Congratulations, your website is being attacked. That means that somebody took the time and effort to:

1. Learn how to launch an attack.
2. Risk prison time.
3. Spend money on a DDoS service or servers, etc.

Your project must be doing great, but at the same time, your website is under attack and unavailable.

Step 2. Enable Cloudflare UAM.

The first step is enabling Cloudflare Under Attack Mode. Cliché, but most of the time, this step is enough to stop any DDoS attack.
Dashboard > Overview → Under Attack Mode.

Step 3. Analyze and mitigate the attack properly.

If Step 2 worked, then that’s great. Your website is back to normal with the inconvenience of a JavaScript page being delivered to all your visitors.

If Step 2 didn’t work, or you don’t want legitimate traffic challenged, then we are in for a bit of manual work. At this point, you are required to build rules to pattern match the attack.

FAQ:

• What happens if the attack has no pattern?

An attack with no visible patterns isn’t a patternless attack.

All attacks have to follow a pattern. If they do not, the results are likely to be senseless and easier to mitigate. There are exceptions, but those are so limited and hard to find that they aren’t worth covering on a more general guide.

• Why do we have to build a pattern?

All HTTP Requests are legitimate unless they are crafted terribly. It is impossible to determine if a request is legitimate or not by inspecting the packet itself. We need to analyze, and group with firewall rules the traffic that we believe is malicious.

Enterprise and automated solutions accomplish this by performing statistical inference and ML (Machine Learning) operations, the process can be more or less complex, but it’s essentially the same for all cases.

Let’s go back to analyzing the traffic. First, go to your Cloudflare dashboard.

If you see a considerable spike in the traffic, you can confirm that you are under a DDoS attack.

If you do not see any spike but, your site is still unavailable, your backend is unreachable to Cloudflare. This can be due to many reasons, none of which we will cover in this guide.

The steps to crafting firewall rules are simple. Follow these steps like a machine, and you will succeed at mitigating your DDoS attack.

• Check if the attack is geo-located.

Dashboard > Analytics > Scroll down to the Requests by country section.

If you see a particular gray area, take note of that.
From our example: Brazil

• Check the source of the requests.

Dashboard > Analytics > Scroll down to the Requests by source section.

Just like we did in the previous step, note down the predominant values.

From our example:

• Path is /forums/

• Referers is the domain itself.

• Ignore the Hosts field. This means that the attacker is focusing their power on a specific subdomain. Pattern-matching this value could lead to an ineffective rule.

• Check the pattern of the requests that the attacker crafts.

Dashboard > Firewall > Overview.

This is the most exciting tab and the one from where we will truly build our firewall rule.

• Check the user agents and write them down.

You are likely to see a set of dominant user agents. Those are the ones that are being used to carry the DDoS attack.

Take a random example from the list:

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3599.0 Safari/537.36

This user-agent belongs to Chrome 71, which is a 4-year-old version of Chrome.

Most attackers are lazy and won’t bother to update nor check whether if the user agent list makes sense or not. Just by blocking this list of UA (user agents), we could single-handily stop the attack.

To prove our point even further, if we look at the second user-agent:

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3599.0 Safari/537.36 we can verify again that the UA is from 2018.

• Check the predominant ASNs (ISP identifier)

In many cases, the DDoS attacks are carried by server vendors that have compromised machines or are rented to launch DDoS attacks.

Another option, even though less common, is that ISP’s ship vulnerable routers that bad actors exploit to launch DDoS attacks. In this case, you’d have 1 or 2 ASNs originating the attack.

In our case, there is no ASN worth noting. We have seen over 3M requests in the last 30 minutes. Pattern matching 365k requests would be redundant given that the UA and path checks match almost 100% of the requests. Only after you have mitigated the bigger picture of the attack, it makes sense to mitigate the smaller sections of the attack.

• Check the HTTP method used.

The most common HTTP Methods used are GET and POST.

In our case, GET is the most notorious request type; we will note it down as it will be helpful in the future when we craft our firewall rule.

• Take a random sample of the already blocked requests.

The firewall logs show insightful information that isn’t shown in the global data displayed in the analytics.

Take a peek at random blocked or challenged requests and look at the following fields:

• Method
• HTTP Version
• Path
• Query String

In my case, there is abnormal behavior, and that is that the HTTP Version is unknown. This is very uncommon and extremely easy to mitigate. The most common HTTP versions for an HTTP DDoS attack are 1.0 and 1.1; there is a reason for this (primarily a limitation on the way the attackers launch DDoS attacks), covering the technical aspects is out of the scope of this guide.

Simultaneously, in this attack example, the query string is empty, but it could be randomized.

To summarize, we have the following:

• Brazil has a lot of presence in the attack. However, it doesn’t group more than 90% of the attack.
• The attacked path is /forums/
• The referer header is the domain itself.
• Approximately 10 user agents are used to carry the attack, as seen in the firewall tab.
• The HTTP Version is UNKNOWN for all requests.

Solution:

• Block requests from Brazil that carry a medium-high threat score, always ensuring that the request is not a known bot.

• Block the requests with UNKNOWN HTTP Version.

• Block the requests that carry the previously mentioned features and have a moderate threat score.

The action:

CAPTCHA is an appropriate choice. I’m skeptical of choosing JS Challenge because it’s “easy” for the attackers to bypass.

If you are sure that this traffic is 100% malicious then, BLOCK is the best option.

Final rule:

(ip.geoip.country eq “BR” and cf.threat_score gt 5 and not cf.client.bot) or (not http.request.version in {“HTTP/1.0” “HTTP/1.1” “HTTP/2” “HTTP/1.2” “SPDY/3.1” “HTTP/3”}) or (http.request.uri.path eq “forums” and http.referer eq “REMOVED” and not cf.client.bot and cf.threat_score gt 3)

Blocking user agents.
I honestly wish that there was a way to create user agent lists and mass-challenge them or block them. In the meantime, we will use the User-agent blocking rule.

Dashboard > Firewall > Tools > User Agent Blocking.

Set the action to Challenge and make a rule for each UA that was part of the attack.

The result.
The attack is mitigated, and the person behind the attack shortly gives up.

Alternatively, the attacker could update the IPs or the attack pattern. Follow the guide from step 0, and you will defeat them again. Cloudflare has unlimited resources, attackers don’t

Final Note.
I will be providing more examples as I get more attacks, I call them frequently, and it doesn’t hurt to show the overall data and actions I took to mitigate them.

[jnperamo]

Update: Mitigating another attack.

Context:
DDoS protection has detected a lot of suspicious requests, and our backend health checks are failing.

Looking at the Cloudflare dashboard, we can see that many requests are being blocked, but still, a lot are reaching our backend system.

To handle this situation, we can take the following steps:

• If you don’t have experience with firewall rules, you can enable UAM (Under Attack Mode) Understanding Cloudflare Under Attack Mode (Advanced DDoS Protection). This will help protect your website from the attack, however, it will add friction to legitimate users and potentially breaks API endpoints.
• If SBFM (super bot fight mode) is detecting requests as malicious, you should enable it. After that, you can add SBFM exclusion rules to ensure that your website’s cronjobs/services are not labeled as bots.
• If SBFM is not enough to stop the attack or is not detecting malicious requests, you should create firewall rules to block the suspicious traffic.

The best way to detect attack patterns remains similar to what we described in the previous message; however, we will see a slightly more complex attack in this example.

Attack overview: (Security > Events).

Take aways from this:

• Rotating IPs: The attackers are using a large pool of proxies that they rotate frequently. It’s not a good idea to flag IP ranges or IPs because we don’t know how many ranges or proxies the attackers have access to.
• Clean IPs: All requests have a threat score of 0.
• SBFM misses: All requests are labeled as “likely Human.”
• Well-distributed: The attack originates from over 10 different countries.
• HTTP/2 protocol: The attack uses the HTTP/2 protocol.
• Targeted path: The attack is focused on the path: “/”
• Randomized query strings: The attack randomizes query strings.
• GET requests: The attack only uses GET requests.
• Randomized referer: The attack randomizes the referer from well-known search engines.
• User-agent strings: All user-agent strings start with “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.”
• The attack appears to originate from around 5 server/proxy-related ASNs.

Considering this, we can implement firewall rules that either challenge or block the attack patterns that are most notorious.

In our scenario, there are several identifiable patterns, such as user agents, paths, and referrer lists. However, since the attack originates from a limited number of ASNs, it’s highly probable that we can stop the attack entirely by targeting this alone. Although the attackers might still be able to change the request method, user agents, and path, it’s unlikely that they can suddenly access new ASNs and servers to continue the attack.

Does this mean we shouldn’t mitigate the other attack patterns? Not at all. We can implement both approaches, where we mitigate the attack patterns and their source. However, if we want to bring back the website online ASAP, the faster mitigation approach is to block those ASNs.

Firewall rule to block user agents targeting path / and method GET

(http.request.uri.path eq "/" and http.request.method eq "GET" and http.user_agent contains "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit")

Firewall rule to block the ASNs that were targeting the website:

(ip.geoip.asnum in {397630 36352 64267 54252 55081 55286})

Attack after blocking malicious ASNs: