Fill lack of capacity in Cloudflare WAF

I have noticed that the WAF in Cloudflare’s Enterprise plan is not that powerful.
He fails to realize that an IP that performs thousands of requests for a type for a URL pattern that generates multiple 404 errors is malicious.

Example:

Checking my access log, I saw that there are thousands of errors to access WordPress directories.

About 10,000 hits in a 10 minute interval that generate error code 404, trying to access the same file in different directories.

Is there any way to create custom WAF rules to detect the following situations?

  1. If the user gets HTTP 404 50x in less than 30 seconds, display a challenge or send it to a temporary blacklist
  2. If the user tries to access the following addresses (in any subdomain) he blocks or displays a challenge:
    • /xmlrpc.php?rsd
    • /wp/wp-includes/wlwmanifest.xml
    • /wp/wp-includes/*
    • /wp-includes/*
    • /**/wp-includes/*

This behavior isn’t enough to classify a visitor as malicious. I see where you are coming from, and I can agree that it can be odd. However, this is not a responsibility of a standard WAF to judge.
Cloudflare WAF validates requests as the edge receives them; it does not have any recursive operations as far as I’m aware.
Adding recursiveness to any process escalates in computing complexity exponentially, and the security gains are minimal.

I believe that Bot Management would detect the kind of behavior that you are describing as malicious.
If it does not bother you enough, some alternatives in the market can work along with Cloudflare and offer the granular filtration you are looking for. However, it’s pricey and typically requires some changes to the client infrastructure.

Subscribers to the Business plan and above can use regexes on firewall rules; this should be easy.

2 Likes

+1 on a solid Bot Management configuration. Or you can follow this example that’s at least ten times my current level of effort:

You are looking for Rate Limiting, not WAF.

If you subscribe Rate Limiting as part of your Enterprise plan, you will get access to the origin response code matching - which allows you to block/challenge traffic that generated a lot of 404/5xx errors.

Here’s a sample:

3 Likes

I suppose the downside, depending on pricing, is that if you’re looking to rate limit based on 404s, but are matching just about all URLs, you’re going to have a lot of “good” traffic you’ll get charged for (if it’s similar to lower plans’ Rate Limiting pricing).

Enterprise customers can negotiate the Rate Limiting pricing with their account team, so the pricing could be different than what self-serve customers get.

2 Likes

Unfortunately, the Rate Limiting price ends up being more expensive than the Enteprise subscription.

You should reach out to your account manager. If you are consuming that many requests, you will get a considerable discount.

Traditionally, that is what using fail2ban on origin server is for - you setup fail2ban jail rules that inspect your web server’s access and/or error logs for regex patterns i.e. file not found 404 errors and via fail2ban jail rules decide what is acceptable i.e. no more than 10x 404 not found errors within 300 second interval and if threshold exceeded, trigger a fail2ban ban action i.e. block at firewall level for non-Cloudflare systems or for Cloudflare systems setup a fail2ban ban action which uses Cloudflare Firewall API to send the fail2ban banned IP to Cloudflare Firewall to be banned at Cloudflare Firewall level.

With fail2ban configured, you can do some of your rate limiting at origin server level and setup fail2ban jail rules to inspect logs for triggered entries for rate limiting messages and then send banned IPs to Cloudflare Firewalll via API.

1 Like

As the requests come through Cloudflare, the IP of these users is from Cloudflare.
I use mod_evasive to block several attempts from the same IP, but I don’t know if activating mod_remoteip I can have the same behavior.

I believe fail2ban also requires the original IP to be the user’s and not Cloudflare’s.

yes that’s why fail2ban implementations with Cloudflare need origin server to configure real visitor IP logging as per CF docs https://support.cloudflare.com/hc/en-us/articles/200170786-Restoring-original-visitor-IPs to ensure logs show real visitor IP.

You can see an example of my rate limiting at nginx origin server level for my Centmin Mod users and inspecting the nginx error log for matches and setting banned IPs to Cloudflare Firewall via API at GitHub - centminmod/centminmod-fail2ban: fail2ban setup for centminmod.com LEMP stack with CSF Firewall

rate limit wp-login.php requests

limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=40r/m;

location ~* /(wp-login\.php) {
    limit_req zone=xwplogin burst=1 nodelay;

Ran Siege load testing again from separate server against wp-login.php to trigger a fail2ban action to Cloudflare’s v4 API

siege -b -c3 -r10 http://domain.com/wp-login.php
** SIEGE 4.0.2
** Preparing 3 concurrent users for battle.
The server is now under siege...
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.58 secs:    7067 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.66 secs:    7066 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.94 secs:  100250 bytes ==> GET  /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 200     0.93 secs:  100250 bytes ==> GET  /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503     0.65 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.46 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.53 secs:    7066 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.93 secs:  100250 bytes ==> GET  /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503     0.60 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.52 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.45 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.45 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.48 secs:    7066 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.94 secs:  100250 bytes ==> GET  /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503     0.93 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.48 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.46 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.51 secs:    7066 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.45 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.45 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.94 secs:  100250 bytes ==> GET  /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503     0.54 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.53 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 503     0.47 secs:    1665 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.51 secs:    7066 bytes ==> GET  /wp-login.php
HTTP/1.1 200     0.94 secs:  100250 bytes ==> GET  /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4

Transactions:                     12 hits
Availability:                  33.33 %
Elapsed time:                   7.82 secs
Data transferred:               0.65 MB
Response time:                  1.75 secs
Transaction rate:               1.53 trans/sec
Throughput:                     0.08 MB/sec
Concurrency:                    2.69
Successful transactions:          12
Failed transactions:              24
Longest transaction:            0.94
Shortest transaction:           0.45

Checking fail2ban log for 149.xxx.xxx.xxx

grep '149.xxx.xxx.xxx' /var/log/fail2ban.log | grep ' Ban '
2017-05-13 03:59:42,227 fail2ban.actions        [11201]: NOTICE  [nginx-req-limit] Ban 149.xxx.xxx.xxx
2017-05-13 04:02:04,713 fail2ban.actions        [11393]: NOTICE  [nginx-req-limit] Ban 149.xxx.xxx.xxx
2017-05-13 04:03:46,051 fail2ban.actions        [11524]: NOTICE  [nginx-req-limit] Restore Ban 149.xxx.xxx.xxx
2017-05-13 04:05:30,268 fail2ban.actions        [11665]: NOTICE  [nginx-req-limit] Restore Ban 149.xxx.xxx.xxx
2017-05-13 05:14:03,388 fail2ban.actions        [11665]: NOTICE  [nginx-req-limit] Ban 149.xxx.xxx.xxx

fail2ban regex nginx error log inspect matches for rate limit message limiting requests, excess

fail2ban-regex /home/nginx/domains/domain.com/log/error.log /etc/fail2ban/filter.d/nginx-req-limit.conf

Running tests
=============

Use   failregex filter file : nginx-req-limit, basedir: /etc/fail2ban
Use      datepattern : Default Detectors
Use         log file : /home/nginx/domains/domain.com/log/error.log
Use         encoding : UTF-8


Results
=======

Failregex: 92 total
|-  #) [# of hits] regular expression
|   1) [92] ^\s*\[error\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by zone "(?:[^"]+)", client: <HOST>,
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [92] {^LN-BEG}ExYear(?P<_sep>[-/.])Month(?P=_sep)Day[T ]24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
`-

Lines: 92 lines, 0 ignored, 92 matched, 0 missed
[processed in 0.02 sec]

Checking Cloudflare’s Firewall Access Rules for fail2ban inserted IP address starting with 149.xxx.xxx.xxx which is remote server I launched the Siege load test from

old example so old dashboard :slight_smile:

2 Likes

Oh as you’re on CF Enterprise, you can do that via Enterprise Bot Management, that’s what I do for my Enterprise plan zone site. Setup a firewall rule which blocks requested files when Enterprise botscore is low enough to be considered non-human request i.e. bots

example rule

(http.request.uri.path contains "/wp-includes" and not cf.bot_management.verified_bot and cf.bot_management.score le 15)

1.2% of requests match the URL path I specified when I hit Firewalll test button (Enterprise plan only feature)

Also as you’re on Enterprise plan your Firewall rules support regex matches so you can use one rule to match many directories via regex

image

Also fine tune by site cookies too if logged in users need separate handling

image

Finally Firewall rules have so many advance fields to match and inspect by including headers and for Enterprise plans also inspect request body too https://developers.cloudflare.com/firewall/cf-firewall-language/fields

2 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.