Under sustained bot attack - Cloudflare Worker Site. Afraid of additional billing

Hi!

I really need some advice here. I kinda accidentally found out that one of my Worker Sites is under a sustained Bot Attack for at least the past 3 weeks or so. Google Analytics is not showing any strange amount of visitors so there was no trigger for me to check the CF Dash. This site normally gets around 300 uniques per day.

In the CF Dashboard I’m seeing request numbers for my Worker (site is completely a worker site - no origin server) that are very high. I now racked up over 50 Million requests in the past 3 weeks. There was a peak that showed 7.1 Million requests on a single day.

I’m really concerned now that I will be hit by a significant usage bill. I think the first 10 Million requests are included (Pro account with Worker add-on). I was ensured by an CF employee on Twitter that ‘Attack data’ will not be billed.

So I took some steps; I enabled ‘Under Attack Mode’ for the site. This seemed to help for a couple of hours (significant drop at first) - but then it started to climb again (back at previous levels of requests now). I also opened a support ticket (#2305801) - about 24 hours ago - asking for help. No reply on that yet (expect for the standard bot response).

Obviously I don’t want to leave ‘Under Attack Mode’ on for too long (bad for SEO and it breaks some functionality on the site as well) - plus - it doesn’t actually seem to help the number of requests that go to my worker.

So my questions are really:

  1. Should I worry about CF charging me for the racked up billable (attack) requests to date? And if this attack continues - should I in the near future? Do I have to do something or proof something to CF to show this is indeed ‘Attack traffic’?

  2. Is there anything I myself can do to mitigate this attack? The sources seem to be hundreds or even thousands of IP addresses so blocking on that doesn’t seem feasible. Anything CF Engineers can do to mitigate this and prevent this traffic to reach my Worker?

Yep, support can help with the billing of that if they can see an attack has/is happening. The billing will currently be $20 (10 million included + $0.5 * 40 (0.5/mil) = $20)

You shouldn’t need to really provide much as they can check however I’d definitely include as much info as possible. Screenshot/data from the firewall, blocked attacks, etc.

Referring to these tutorials is a good start:

7 Likes

Thanks Walshy! I’ll go through the linked articles and see if there is anything there that can help me.

I believe that you will be billed unless you report it to Cloudflare. @Laurie is part of the billing department; I feel like her input would greatly value this scenario.

Follow the guide Walshy linked, if you need help then I’d ask the following information:

  1. Photos of the visits analytics (as a whole).
  2. Photos of the WAF analytics and a random sample of events that were automatically blocked.

From this, I will be able to more or less build a profile of the attack and guide you further, or even build a firewall rule to stop the attack.

I believe that considering that you are in the PRO package and that we have some people in the community with a lot of experience in DDoS mitigation, it would be best if you closed the ticket.

Be advised that the CF challenge (UAM) does not affect legitimate bots, they are able to freely go through it and skip it.

2 Likes

@jnperamo - Hey Thanks! With regards to closing the ticket - I’d like to keep it open especially for the billing part of this. With regards to SEO - Under attack mode causes a delay in loading which kills web vitals… which is bad for SEO I guess? Also - it scares away customers.

In the mean time (also based on the info in the articles) - I managed to find out - at least from recent requests - that all those requests shared a single User Agent and were all coming from the US. So I now created a firewall rule that block requests from the US with that specific user agent.

So far so good.

As requested, here are some data from the Dash. As you can see - there is a specific UA there that caused a lot of the traffic. Now blocked.


Screenshot 2021-11-17 at 18.53.12

Perfect! I’m glad that the guide was useful, I was going to point out exactly that. Can you share the complete user agents? Just so that we can verify that they are safe to block.

What I’d trigger in the firewall rule is the following:

  1. ASN Number
  2. User agent(s)
  3. Country

Honestly, if you don’t want to receive traffic from that ASN, I’d block it from the root and nothing else.

Yeah, this is the major downside of the JS Challenge :frowning:

2 Likes

I’ve seen traffic also coming from other ASN’s. The IP ranges hitting it are huge. But - it does seem the UA is always the same (although I can’t be too sure so keeping an eye out now).

The UA is: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36

I checked my own analytics data and for the past 3-4 months or so I had only 30 or so legitimate visitors with that UA combined with Country US. I can accept that maybe a very small number of real customers are blocked now.

2 Likes

Sounds good; I’d also keep an eye on the HTTP Version used in those requests and perhaps other headers that they might be missing and are generally present in a legitimate browser.

1 Like

That’s a good tip, thanks!

My new FW rule blocked 7K requests in the past 30 minutes… seems to work for now.

2 Likes

Blazingseo seems to be a website scraping service!

4 Likes

Yes I noticed that as well. Someone might be using their service to scrape the site? Not sure why though?

It could be someone scraping your site for that purpose or using their service as a means to attack or flood your site with requests. I’ve seen bots try to make exactly 100,000 to 102,000 requests to my site that is running a CF Worker. I guess they’re trying to exhaust CF Worker free plan users quota of 100K free Worker requests to take down a site. They attempted this for over 4 months! Luckily, I am not on CF Worker free plan.

That’s where Cloudflare paid plans are useful as you get higher Firewall rule quotas and more Firewall features as you go higher on your Cloudflare paid plans. CF Business and Enterprise plans also get Firewall rule regex matching support so you can have finer grain control over what your Firewall rules match and Cloudflare Enterprise plan has Enterprise Bot Management which can further block such automated non-human requests.

@Walshy posted links for DDOS mitigation steps are very useful.

Outright blocking of ASN = 397630 would be useful here. I already added them to my Firewall rule for blocking known scraping services :slight_smile:

4 Likes

I ended up also blocking that ASN# completely. Now I hope that CF will not bill me for the usage. Hopefully @Laurie can put my mind at ease.

1 Like

@Gloria is on right now and might lend a sympathetic ear regarding Workers billing when DDoS’d.

4 Likes

I did find the ticket and I’ll check to see if there should be any charges and I’ll update there @michiel

Thank you.

4 Likes

Psst! When you hear back, can you let us know in general what they said? This has come up several times and it’d be nice to know how much slack they cut you in billing for an attack.

3 Likes

So far it doesn’t look good. According to @gloria I would still need to pay for the additional 40 Million requests. I don’t understand the motivation behind this really. It seems there is a pretty clear stance that customers do not have to pay for usage racked up by attacks? This is also the position of multiple CF people I spoke to on Twitter.

Now it doesn’t really break the bank - however this rigid position does make me have to worry more about using CF. Which is a shame as I’m paying for this product to accomplish the opposite - peace of mind.

I might ping some CF people I have in my network to get their opinion about this as well.

I don’t know the site in question and how the Workers Site is set up, but have you thought about moving it to Pages where you can be hit like this without worry due to the unlimited nature of the product?

1 Like

Hmmm, thanks - that might be worth investigating indeed.

1 Like

That’s unfortunate. I’m guessing that Cloudflare only skips from the bill the traffic that the system automatically deems malicious. However, It will be best to wait for a final verdict on the case.

Part of the problem (IMHO) is that sometimes DDoS Attacks aren’t adequately defined.
24/7/365 DDoS Attack Mitigation SLA, for example, does not typically include HTTP attacks; this policy can be contradictory to some clients that expect mitigation against any attack that makes their website/service unavailable.

Be advised that this is an industry standard. SLA against HTTP Attacks is only given by contracts that involve ridiculous amounts of money or companies that are likely overestimating their capabilities.

3 Likes