DOS attack by Cloudflare against our website - no support provided

Website, Application, Performance DNS & Network
1

Hi,

I raised a ticket with Cloudflare 18 days ago - case 2614256 - because we affectively have a denial of service attack against our website from Cloudflare - it has knocked out our website multiple times now.

We are receiving 10+ requests a second with the following data:

2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 162.158.191.188 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 285
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 162.158.51.130 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 305
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 162.158.74.31 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 41
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET /gb/contact/ - 443 - 172.70.175.236 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/86.0.4240.75+Safari/537.36 - 200 0 0 288
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 172.69.156.204 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 334
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 172.71.166.2 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 277
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 172.70.182.144 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 387
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 172.71.130.17 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 49
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 162.158.147.156 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 447
2022-11-27 23:59:55 W3SVC4 10.0.0.240 GET / - 443 - 172.68.230.138 Mozilla/5.0+(compatible;+Cloudflare-Traffic-Manager/1.0;++/traffic-manager/;+pool-id:+8a8fc0633edaca60 - 200 0 0 352

All that Cloudflare support have said (after multiple requests for support) is that the pool id isn’t owned by our account - but that’s it - no answers.

The web hosting company is, as you can imagine, not very happy with us or Cloudflare!

Any advise?

Thanks

Martin

2

I don’t really call this a “DoS attack” since it’s just a normal health check traffic sent from the Cloudflare Load Balancer and you mentioned that it’s just 10+ requests per second, which is… very low.

Anyway, it seems like someone configured their load balancer in Cloudflare to include your server within their load balancing pool. Are you using a shared hosting service where one server is being used by multiple clients, or do you have your own dedicated server in your web hosting provider?

2 Likes
3

Yeah… to be honest - we’ve had a case open with Cloudflare for 18 days and are so sick of not getting a response we thought we’d put it here.

The 10+ reqs / second is still high for a single site with a single load balancer rule and a single health check on it - both are set for 60 seconds so there shouldn’t really be this amount of traffic coming from Cloudflare even with checks coming from multiple locations. Additionally we don’t see this level of traffic going to other sites configured in the same way.

The server is, as far as I know because we don’t host it ourselves, multi-client but the log specifically mentions our website uri.

Cloudflare support have said that the request is coming from a pool that we don’t own so what you say about “someone” (other than us) configuring their load balancer to point to our site adds up.

10+ sec isn’t big - but it’s unnecessary traffic being passed to our web host that they’re complaining about. And, for whatever reason as I don’t have access to the servers, the site has been knocked out multiple times to the failover site which is our biggest concern and the hosting company is pinning it on this.

All the best

Martin

4

Hi Eric,

I’ve tried replying but the message has bizzarely been caught as spam?? I can’t see anything in there that could be construed as such but hopefully they’ll release the message soon!

I didn’t want you thinking that I hadn’t replied.

Thanks

Martin

5

Eric,

I’ll try again…

We posted here because we’re not getting any useful response from Cloudflare support (after 20 days, we’re just getting fobbed off each time and dropped).

We should not, even with multiple region checks, be getting this many hits from Cloudflare monitors.

Additionally - we’ve been informed that the pool id for the checks doesn’t even belong to our account. Regardless of whether this is a very low amount of traffic or not we’ve been informed by the hosting company that the site has become unreponsive - and they are attributing this to these monitor checks.

Have you (or anyone else) seen an instance where Cloudflare monitor checks, from an account that you don’t own, can reach a level that it can affect performance or availability of a site?

Is there a way to identify which account owns the pool id doing the checks?

Cloudflare support did come back yesterday and ask us to disable a health monitor check which we did, but this made no affect to the amount of traffic hitting the site from Cloudflare monitor checks as seen in the initial post.

Thanks

Martin

6

The User Agents seem a bit off compared to my CF Load Balancer reported user agents so not sure if it’s a valid one. Are you yourself using CF Load Balancer at all?

My User Agent doesn’t list with +pool-id and ++/traffic-manager/. Is the additional + in yours just the way you have configured your origin server logging or something?

i.e. my CF Load balancer related requests look like this without that many additional + in some references = Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: xxxxxxxxxx)

Depends on your health check configurations for check interval and regions which can drive up traffic. For instance if on CF Business or higher you can set check intervals as low as 10 seconds and set check regions to all, so 250+ data centers and 13 regions with each health check coming from 3 datacenters per region is at least 39 datacenters doing multiple by checks every 10 seconds can add up. See https://support.cloudflare.com/hc/en-us/articles/4407016052493#health-checks

Anyway CF Health checks have a different useragent anyway with referenced healthcheck-id https://support.cloudflare.com/hc/en-us/articles/4404867308429

Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; healthcheck-id: <HEALTHCHECK_ID>

Though confusingly at Manage monitors · Cloudflare Load Balancing docs

Each health check has the HTTP user-agent of "Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: $poolid)" , where the $poolid is the first 16 characters of the associated pool.

It’s more likely like @erictung stated that another CF customer misconfigured their load balancer origin IP to your IP i.e. typo in IP address. Not much CF can do really. I once had a porn site misconfigure their DNS records and pointed to my forum’s IP address LOL. So all those porn site visitors landed on my non-adult forum :slight_smile: IIRC, it was around 2,000+ requests/second that hit my forum and my forum handled it fine :slight_smile:

Any decent web host should understand that this is outside of your control. If they can’t handle 10+ requests/sec, I’d be looking for a different web host as even legit traffic could easily surpass that rate of requests.

Only thing you can try is get CF support to lookup that pool id 8a8fc0633edaca60 and contact the CF customer concerned. In meantime, you can just block the user agent matching 8a8fc0633edaca60 on your origin web server side.

Edit: an additional more involved option is to setup customer client SSL certificate based Cloudflare Authenticated Origin Pull certificates at zone or per hostname level (the latter 2 links below)

This involves you creating your own CA root and optional CA intermediate SSL certificates and using them to sign your own created SSL client certificate and setup your origin server to only allow traffic from requests then present your created SSL client certificate that you upload to Cloudflare via API.

So any other CF customer’s misconfigured settings for DNS and/or Health check/Load balancer based requests will be blocked from your configured origin server as they are missing that custom client SSL certificate you created and signed using your own CA root/intermediate SSL cert - so only legit requests from your Cloudflare domain zone will hit your origin servers.

Example of how I setup custom CA/client SSL signed SSL certificates with Cloudflare Authenticated Origin Pull for per hostname setups using Cloudflare API at GitHub - centminmod/cfssl-ca-ssl

2 Likes
8

First and foremost - thank you so much for taking the time to reply. That’s all really useful information.

I believe the + symbols in the log data will be being generated by IIS (I can’t see any other reason why they’d be there as I don’t see them in my other Apache or Nginx logs).

I can confirm that we are using a load balancer for that site (for failover) but the checks are configured to every 60 seconds. Logs from a different server that are behind a cloudflare load balancer show traffic manager requests every 60 seconds as expected.

Blockquote If they can’t handle 10+ requests/sec, I’d be looking for a different web host as even legit traffic could easily surpass that rate of requests.

We’ll see what they say :slight_smile: they’ve been good before so we’ll see how it pans out.

I’ll go back to Cloudflare and ask them if they’re able to identify the owner of the “bad” pool id but won’t hold my breath.

Again, I really appreciate the answer and information.

All the best

Martin

10

Hey there,

Sorry for the trouble you’re having with this traffic.

I have replied on your ticket with instructions that should help clear this up.

2 Likes
11

Scott

You’re wonderful - many thanks!!

Martin

2 Likes
12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.