Mutli-domain DDOS prevention

Cloudflare seems to think only in terms of 1 domain features. What if you have hundreds or thousands of domains? What if a DDOS attack is running down a list of those many domains they likely got from a nameserver database?

Cloudflare recently allowed a massive swarm from Google Networks to hit hundreds of domains simultaneously, which cloudflare knows the end-point is a single IP, and can see there were thousands of 404s being returned, but just let it all ride on through.

I would imagine cloudflare, to live up to its own standard, would likely monitor DDOS attempts not per-domain, but per destination IP, since its their OWN PROXY that facilitates the DDOS. I could not block the attack because it was CLOUDFLARE relaying it.

What would Cloudflare expect, that I manually go click “under attack” across hundreds of domains?

This is a pretty clear shortcoming I would think?

Hi there!

I am really sorry to hear about the malicious traffic you were experiencing.

Unfortunately I can’t really speak to the details of the event you are mentioning either because I wasn’t there, but I do hope I can give you some information that may help in the future for applying security settings more broadly.

You are correct that many of the CF security settings are enabled/configured per domain (exceptions to this include some features like IP Access Rules which can be applied across every zone in your account) – and the CF dashboard isn’t really an exception, meaning that while some settings can be applied globally across your CF account (e.g. IP Access Rules) it too focuses on applying these settings specific to the zone/domain.

In cases of applying something more akin to “Bulk” changes, the Cloudflare API is going to provide our best option for being able to quickly apply changes across multiple sites in your account.

By writing a script in something like Bash or perhaps using some programming language-specific tools from our open source page, you can, for instance automate applying “I’m Under attack Mode” to domains using what essentially becomes a series of calls to our API endpoints.

I should however since you said “thousands” clarify that we do apply some limits on our API, specifically:

The Cloudflare API sets a maximum of 1,200 requests in a five minute period.

So you would just need to make sure that your requests stay under this threshold to make sure our API actually successfully completes the goal you are trying to accomplish.

While I understand that this may not be quite what you are imagining, and admittedly, it does require some considerations and development work, I really hope this information can help provide an option that works for your use-case.

Best,
Peter

6 Likes

Hi thanks for the response Peter. We are fully aware of the API and have integrations there already, but as you noted, your limits would not really enjoy that kind of bang on your API.

I would just think that since your proxy facilitates an attack, you would endeavor to be “aware” of the source server IP and then also aware of the many requests happening to it, and then also aware then of when that looks suspicious.

That would of course be a far better attack prevention system then a per-domain setup.

As it is now, I would have very little choice but to just block cloudflare, which then of course leads me to just stop using your services because they are facilitating DDOS attacks without any real means of handling it.

Hi there,

Sorry I wasn’t able to offer any new information there – it sounds like you are already well-acquainted with our API.

I apologize if I might be misunderstanding here, but I think I might need some clarification on what sort of attack you were seeing – without providing any personally-identifiable information (for security and privacy reasons), are you able to confirm if this was traffic from clients you are seeing being proxied from CF’s edge to your origin servers or was this something else?

Best,
Peter

4 Likes

Its about 400 concurrent connections that apache reports as coming from 66.249.64.0/19 Google network. The firewall reports its from various cloudflare ips.

Apache logs show its systematically hitting one domain after another and MOST of the hits are to super strange urls like /l/J/10.1b?1a=11&f=6/g/14/2c.j&19=Z and /g/F/E/2G.D and /d/b/l/k/

Which are nonsense and can’t possibly be spidered, which leaves me to believe either googlebot is going bonkers, or there’s someone compromised a bunch of servers on their network thats using a DNS list to DDOS. Though why DDOS against such strange urls which typically form nothing but a 404 which servers typically handle really well. In this case it happens to be against a website framework that parses /any/url and looks for matching content.

I wrote network abuse at google but have not yet received any reply.

Should be something cloudflare builds against so they aren’t helping DDOS attacks that we admins on this side have very little power to fight without getting rid of cloudflare entirely, or atleast the proxy aspect of it.

Hi there!

Thanks so much for your reply,

Just a quick clarification on your point here:

that apache reports as coming from 66.249.64.0/19 Google network. The firewall reports its from various cloudflare ips.

Was the firewall you are referring to here the CF Firewall or your origin?

Also, apologies in advance if this doesn’t work for your use-case, but do you ever expect legitimate traffic from cloud providers?

If not, one easy thing you can do to filter out the human traffic is to apply a Captcha Challenge to requests from these cloud provider’s ASNs using either IP Access Rules or Firewall Rules.

You can also further augment this condition if doing this in Firewall Rules to make use of cf.client.bot field so that you can make sure the search engine’s crawler is not getting blocked.

( Here’s a link to our FW “Fields” page in our dev docs in case it helps any: https://developers.cloudflare.com/firewall/cf-firewall-language/fields/ )

I should mention that if you apply a Captcha Challenge (verses outright blocking the requests) then it will make sure that anyone who might have set up their own VPN endpoints on any of these cloud provider to reach your site is still able to pass through (provided they can solve the Captcha) – but it should help to curb automated traffic.

Do you think this might be something that will work for your use-case?

Best,
Peter

2 Likes

Origin firewall.

I dont feel captcha is an appropriate way to deal with filtering bad bots. When you run marketing websites, and want to kill your optins and sales, then sure, use that captcha. There is also a large growing number of people using VPNs, as well as plenty of healthy bots you want to spider your sites.

Taking such actions for the exception of these kinds of attacks seems excessive especially when the answer is simple: block the abusing CIDR. That of course is easy to do when you control the access, but we’ve given cloudflare that power, and then unfortunately cloudflare doesn’t provide the tools to effectively deal with it across hundreds or thousands of domains.

You can of course run your business however you want, but I would think these things are obvious to implement:

  1. pay attention to traffic patterns routing to single origin IPs no matter what the domain is.
  2. provide global tools to create firewall rules or such that apply to all domains in the account, without having to loop through them all applying to each one individually.

We need the ability to react to DDOS account-wide, especially since cloudflare achilles heel is that you frequently use the same NS per account (with some rare exceptions) and there are databases out there cataloging domains per NS.

This way, attackers can dodge your prevention by simply focusing their attack on those nameserver lists, making cloudflare complicit in their DDOS.

To me the solution seems obvious and its not ‘use the api’ - if you want to harden your service and stop attackers from circumventing your measures, you simply have to implement at least #1.

Hi there,

You are indeed correct that applying a Captcha Challenge to an entire ASN/IP Prefix would probably not ideal for search engine indexing, and this is partially why I wanted to mention using CF’s Firewall rules combined with an exception for cf.client.bot to prevent these crawlers from getting blocked.

A list of bots we try to identify with this operator is here:

This would still Captcha Challenge VPN users using that cloud provider IP prefix/ASN, so there are trade-offs going this route that I wanted to make sure I call out first.

That being said, regarding your comment:

answer is simple: block the abusing CIDR

While this next option somewhat tosses out the FW rule option I mentioned above – we actually have this option available for you today via IP Access Rules. While our CIDR support here is somewhat limited (I think we support /16 and /24 rules for IPv4 and /32, /48 and /64 for IPv6) – you can actually block/challenge an ASN too for all zones in your entire CF account. for instance AS15169 is Google’s – applying a temporary challenge during a period of unexpected traffic across all your domains, is as easy as adding the desired ASN you wish to challenge and then selecting "All websites in the account" – then the rule will be applied globally across all your sites with no need to switch Dashboards.

I hope this helps!

Best,
Peter

3 Likes

Oh I just understood that while “firewall” is under a domain, you allow settings in there that can apply to all websites in the account. Perhaps your UX guys can improve on that - never occurred to me to even look inside a domain’s settings for the ability to apply rules across the entire account…

Yes that feature helps in these situations, but the lack of full CIDR support is restrictive, but workable. Thanks for the tip.

Still think cloudflare might want to consider at least option #1 as it greatly enhances your mission to prevent abuses like DDOS if you look at origin IP instead of ‘domain’ traffic on its own.

This topic was automatically closed after 30 days. New replies are no longer allowed.