Cloudflare flooded my server!

bug
#1

I noticed a user that is downloading a big tar.gz file (~800MiB) using multiple threads. However, the download has now taken hours and CF has requested more than 50GiB for the same for the user with the same IP.

Doesn’t CF support byte ranges? If not, this becomes a DDoS type scenario very easily.

Looking at the CF stats/Analytics page, the amount of data served does not match the amount of data requested by CF from origin server.

0 Likes

#2

I looked at the logs. Cloudflare requested about 100GB during 11-13pm. However the analytics page says only 396MB was served for the same period.! Surely this must be a bug?


Compare with the bandwidth log on origin.

It was only the one file downloaded by multiple sessions in parallel from the origin by the cloudflare server.

0 Likes

#3

I’d really like to find out what happened from cloudflare point of view. Is there someone from Cloudflare here that can help?

Thank you.

0 Likes

DDOS mitigation
#4

Same problem, cloudflare flooded over 1TB which cost me $60 bandwidth charge in 1 f**king night




0 Likes

#5

do you have any logs that show who and what happens?

0 Likes

#6

Do you @user605 @user8273 enforce firewall in your machines to only accept traffic from cloudflare and authenticated origin pull?

0 Likes

#7

All traffic came from Cloudflare IP. Somehwere around 50-100 concurrent requests to the same file. This is why I concluded the end user was doing a multithreaded download and that cloudflare, which doesnt cache large files, downloaded a full copy for each of the threads.

Some excerpts from the log. 162.158.111.221 is the CF edge IP.

mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:31:59 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:30:09 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:31:58 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:34:45 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:34:13 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:33:40 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:34:13 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:31:27 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:31:26 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:32:33 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:35:17 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:33:40 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:15:43 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:15:12 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:32:32 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:33:07 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:36:19 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:34:45 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:38:27 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:29:37 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:35:48 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:14:39 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:37:23 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:38:57 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:39:28 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:37:52 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:37:52 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:37:21 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:35:47 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:36:18 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:38:26 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”
mirrors.tnonline.net 162.158.111.221 [21/Feb/2019:12:38:59 +0100] “GET /haiku/haiku-release/r1beta1/haiku-r1beta1-x86_gcc2_hybrid-anyboot.zip HTTP/1.1” 200 978156086 “https://www.haiku-os.org/get-haiku/” “Mo
zilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”

0 Likes

#8



My vm just been deleted for overcharged fortunelly i have previous backup

0 Likes

#9

My theory thus far is.

  1. a user wants to download a large file, which doesn’t get cached by CF.
  2. user experience a slow download so he initiates a multithreaded download, where each thread is only a few MB.
  3. each thread downloads using http byte ranges.
  4. a fault in CF causes CF to request the full file from origin server instead of the specific byte range
  5. CF does this for each thread /request that the user does.
  6. result is that one user downloads 1GB from CF edge server but CF downloads 1GB x n threads from origin. In my case about 120GB.

Clearly this is Cloudflare flooding and not the end user.

0 Likes

#10

Does anyone have any ideas on this and how to prevent this type of problem?

0 Likes

#11

I don’t know, but I am pretty sure anyway you shouldn’t use cf with large files(I don’t want to speak in their name so the best will be to contact them about it), but if you want easy cheap solution you can use something like b2 storage for the large files https://www.backblaze.com/b2/cloud-storage-pricing.html

0 Likes

#12

Thanks. Not all files are large. I don’t want to host elsewhere. In any case I don’t think it’s right that Cloudflare should increase bandwidth usage a hundred fold or more. Do you ?

0 Likes

closed #13

This topic was automatically closed after 30 days. New replies are no longer allowed.

0 Likes