How to protect cloudflare worker from ddos?

I try to loadtest my cloudflare worker url using loader.io.
And try 10000reqs per sec. It still going thru without any error. So bad guys can just do ddos and i will be charged huge sums of money? because i’m in pad plan now.

How to protect our cloudflare worker to such attack? I thought cloudflare http flood protection will kick in. but its not at all.

unfortunately its not.

free gets 100k free requests per day
unimited gets 10mil free request per month (2)

  1. You will automatically be billed $0.50 per additional 1 million requests.
  2. You will be automatically billed $0.50 per additional GB of storage, $0.50 per additional 1 million read operations, $5.00 per additional 1 million write operations, $5.00 per additional 1 million delete operations, and $5.00 per additional 1 million list operations.

Keep in mind that Cloudflare might already be white-listing loader.io IP’s because users are already using it to load-test their Workers (I’ve been doing that for more than a year). So why would they categorize it as an attack then?

You’re doing layer 7 application level attack so need to use layer 7 tools CF provides like CF WAF, Firewall rules & CF Rate limiting. Layer 7 mitigation can’t be 100% automated as CF can’t 100% know if the requests are legit or not without you telling it via hints i.e. CF WAF/Firewall Rules, Rate limiting.

You can use rate limiting to protect workers but CF rate limiting cost more than CF workers so might as well let CF worker bear the brunt of it if you have CF worker which does caching to protect your origin. Of course if your CF worker is not setup for caching to offload work from origin, then you’ll incur CF worker costs and still overload your origin

  • CF rate limit = $50 for 10 million good requests
  • CF workers = $5 for 10 million worker requests

You can also setup fail2ban on origin server and configure it to talk with CF Firewall API so fail2ban jail rules you specify for bad request type traffic gets banned and the IP ban gets sent to CF Firewall via API to ban at CF Firewall level.

For instance if your origin Nginx server is setup with rate limiting of say 10 requests/s to /register.php link for same IP it will log rate limit log entry in your nginx logs. If you setup fail2ban to read that nginx log looking for that match, then you can ban that IP that hits 10 reqs/sec to /register.php and configure fail2ban to talk to CF Firewall API to pass on that banned IP to CF Firewall which will ban the IP at CF edge server.

6 Likes

Nice theory. As soon as I read it. I try to do load testing using my home computer with apache bench.
But it still doesnt block it even at 10krequest per seconds.

I’m still curious what is the purpose of cloudflare’s ‘http flood’ firewall rule?
I test it again targeting 3 wordpress site which use cloudflare.

i try it like this: https://www.wordpresssite.com/?s=[random-number]
as soon as i try loadtesting 10k/sec all those website is down. You can try it to any wordpress site which use cloudflare.

Known malicious traffic patterns versus unknown. Cloudflare wouldn’t know if it’s legit without your telling it so via hints i.e. CF WAF/Firewall Rules, Rate limiting

Are you using CF free or pro plans ? CF WAF is for paid plans only.

CF WAF has some preconfigured WAF rule sets for specific web apps too like Wordpress

1 Like

its a lot of work. but its really genius using fail2ban as a free rate limiter.
now cloudflare seems useless if i need to do like this

Not useless, fail2ban just passes the ban IP to CF Firewall so that CF edge servers eventually do all the work in banning that IP at CF edge server level. CF has 200+ datacenters worth of CF edge servers to handle large layer 7 DDOS attacks which will have more computation and network bandwidth than your single origin server would be able to handle via fail2ban alone :slight_smile:

Cloudflare provides the tools for you to use, just need to use them optimally :smiley:

FYI, I’ve seen large forums handle such layer 7 DDOS attacks with such a fail2ban + CF Firewall implementation with ease for 1-3 million requests/second ! Find me a DDOS mitigation service able to handle that for free to US$20/month :slight_smile:

CF Firewall custom rules I tailored for my forum sites has virtually wiped out brute force login attacks as I create custom CF Firewall rules that target my login/registration pages for certain criterias i.e. ASN, IP, Country, user agents, HTTP request protocol etc = me telling CF via hints what is legit or not legit traffic for my particular web app :slight_smile:

5 Likes

@eva2000 just curious, say your fail2ban/cloudflare service hiccups during the ‘actionunban’. then technically, that IP wouldn’t get unbanned and that firewall rule on cloudflare would remain. just curious if that edge case was considered?

fail2ban has a status command and I also wrote an extended status wrapper script for more details for each fail2ban jail rule so you can inspect how old a banned IP is. Then I have a separate custom CF Firewall API script which can do bans/unbans by IP, IP ranges and even unban based on IP ban age. So you can script purging of banned IPs from CF Firewall via API for say older than XX seconds age as a way of cleaning up CF Firewall banned IPs that come from fail2ban or any other app (because custom script can purge based on CF Firewall’s IP comment entry to target specific types of banned IPs i.e. match with fail2ban banned IP’s CF Firewall comment entry).

i.e. unban all Cloudflare Firewall banned IPs older than 5 seconds

./cf-firewall-api.sh unban-age 5                    
1.2.3.4 ip 1592016378 1592021977 5599
1.2.3.5 ip 1592016380 1592021977 5597
1.2.3.6 ip 1592020117 1592021977 1860
1.2.0.0/16 ip_range 1592021713 1592021977 264

unban 1.2.3.4 older than 5
{
  "result": {
    "id": "da32cd39c05640d298c306478a745bed"
  },
  "success": true,
  "errors": [],
  "messages": []
}

unban 1.2.3.5 older than 5
{
  "result": {
    "id": "d426a1a999f14e09b88ef78db150d6d1"
  },
  "success": true,
  "errors": [],
  "messages": []
}

unban 1.2.3.6 older than 5
{
  "result": {
    "id": "0a3e4d3b866c4777b662546535a2ef11"
  },
  "success": true,
  "errors": [],
  "messages": []
}

unban 1.2.0.0/16 older than 5
{
  "result": {
    "id": "9ae9ae6d41c24c738a5bb34cbffaaed7"
  },
  "success": true,
  "errors": [],
  "messages": []
}

@eva2000 fail2ban on your hosted server is not really solving the problem like @ray9 hinted, you just creates a new bottleneck.
your fail2ban server will easily fail under big load and then you are back to the same problem

Yes fail2ban isn’t the whole solution, it’s only part. That is why I originally stated in below quote

and

@eva2000, care to opensource your cf scripts? that looks useful and could be very helpful for people implementing these fail2ban solutions. I’m currently considering it in my environment.

1 Like

Majority of scripts are mainly for my use or my paying clients and specifically tailored to my own Centmin Mod LEMP stack users which use CentOS and CSF Firewall (wrapper to iptables).

1 Like

First time I hear of a full top-to-bottom LEMP stack, seems really interesting.
How long have you used it for clients?

Also, why LEMP and not something like ISPConfig?

LEMP stack what I am familiar with that’s why. I develop Centmin Mod for performance and scalability as I spent over a decade working for vBulletin forum software and working with some of the largest community forums online to ensure their forums scaled/performed. So took what I learnt and put it into Centmin Mod LEMP stack :slight_smile:

Some of the largest internet properties online use Centmin Mod LEMP stack including Alexa Top 5,000 and 10,000 sites as well as 10% of all the largest Xenforo forums communities online https://community.centminmod.com/threads/centmin-mod-lemp-powers-10-of-xenforos-largest-forums.16435/ and some of the largest vBulletin and Invision Power/Board forums :smiley: Currently, there’s between 2,000 to 3,000 new Centmin Mod installs per month for past 9 years and growing :slight_smile:

Obviously, a lot of those web site/communities leverage the wonderful Cloudflare product/service offerings that definitely help :sunglasses: As you know alot of forums do get DDOS attacks so Cloudflare does its part to help :slight_smile:

3 Likes

Whoa! I had no idea you where a sysadmin wizardess!

I’ve hosted quite a few PHPBB forums in my days, but never got it to scale well so the past 5 years it’s been NodeBB instead. Considering how far Nginx + PHP have come, I’d expect performance to be pretty much on-par now.

2 Likes

Yeah that’s my world, wordpress/community forums and server backends. Pair that with Cloudflare and you have a wonderful performant base thanks to Cloudflare’s product/service offerings :slight_smile:

PHP has come along way, my PHP 7.4 vs 7.3 vs 7.2 vs 7.1 vs 7.0 benchmarks https://community.centminmod.com/threads/php-benchmarks-7-4-vs-7-3-vs-7-2-vs-7-1-vs-7-0-php-fpm.18741/ :slight_smile: and how using PHP with Profile Guided Optimizations can boost Wordpress performance https://community.centminmod.com/threads/php-7-3-vs-7-2-vs-7-1-vs-7-0-php-fpm-benchmarks.16090/#post-69010 :slight_smile:

3 Likes