Concurrent Requests fail through Cloudflare

Website, Application, Performance
1

Hello Cloudflare Community!

I’m doing Concurrence tests with my server and, they work internally as I have worked with pm.children parameters, along with nginx limits so the test gives successful resources. And it does show 100% successful requests.

With Cloudflare however, I get 429 in the middle of the execution of the test, so I wonder if Cloudflare internal queue is not processing requests fast enough, giving them to my server so I can respond them correctly. Basically, finding the bottleneck.

I’m running 300 concurrent requests.

2

May I ask if you are trying to do a cURL request to your domain/Website behind Cloudflare, or rather to some other domain/Website which is using Cloudflare? :thinking:

You might also be hitting some Rate Limiting Rules which owner of that particular Website has configured with specific parameters (time-frame, same IP, frequency, requests …):

Can you share some screenshot of this error?

1 Like
3

You might be trigger some CF default DDOS mitigation processes. For load testing on CF might want to start with Simulating test DDoS attacks · Cloudflare DDoS Protection docs

1 Like
4

Sure @fritex, I’m using Jmeter to do the concurrence tests and, I’m testing my website which has Cloudflare configured as a Middleware, which is why I get the specific 429 error from Cloudflare.

Here you can find a snippet of one of the error codes:

Thread Name:Concurrencia Test-ThreadStarter 1-260
Sample Start:2022-10-31 11:08:28 WET
Load time:99708
Connect Time:124
Latency:99708
Size in bytes:531
Sent bytes:319
Headers size in bytes:531
Body size in bytes:0
Sample Count:1
Error Count:1
Data type ("text"|"bin"|""):
Response code:524
Response message:


HTTPSampleResult fields:
ContentType: 
DataEncoding: null

The ResponseData is empty and, the response headers are:

HTTP/1.1 524 
Date: Mon, 31 Oct 2022 11:10:08 GMT
Content-Length: 0
Connection: keep-alive
Cache-Control: no-store, no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=MDCfqc7nTOe37LrI3wRi%2BLdcI4o7jV6Q5XTR8nVGzK2WmGfEsN7RA4yHqRhsJ304f0IAczlVCmNRch6ibkCViAlh7m0JCxT0lJzir9vzom0i%2BYbWScypLfiT6aVDg%2F0w5MvT"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 762beedb3f09f130-CDG

I do not have any Rate Limiting Rules and, I don’t see any documented ones regarding the tests that I’m doing. And in the case that I’ve getting blocked by any of these, I would be getting a Retry-After header as I see in the Configuring-Cloudflare-Rate-Limiting document, which I’m not.

@eva2000 Are there any ways of telling if I’m getting blocked by this limiting?

As my test descends into doom (starts with one error, then 3, then 15, then all the requests are error), my server through Cloudflare is unavailable. But I do not get this behaviour when testing directly as I get all successful requests.

Even when I do the test through Cloudflare and I get a lot of errors, I can see that after some time, my server did responded all of the requests, so they are all marked with a 200 status.

5

Point the load tool directly to your origin? There’s little useful information you are going to get by trying to load test Cloudflare… it handles 4m+ https requests per second.

A 524 error happens when your origin is unhappy. How are you measuring load on the origin and how are you scaling to meet it?

6

I’m scaling my server with rate limiting settings in nginx and more pm_children in php, so the requests are faster. Though this for some reason doesn’t affect Cloudflare. I’m measuring al via tests and, once I get 100% successful responses in my origin, I stopped scaling any of these settings.

Though at this point it would be better to do as you said, point directly to origin and forget Cloudflare internal Queue, at least for this concurrence test.

7

If you have Cloudflare Pro or higher plans, the Firewall analytics Security Events · Cloudflare Web Application Firewall (WAF) docs will show more details including CF DDOS/WAF mitigation triggers that are native to CF including DDOS managed rules Managed rulesets · Cloudflare DDoS Protection docs

i.e. HTTP DDOS Protection Managed Ruleset

For my load testing I’ve sometimes tripped CF’s DDOS HTTP attack managed rulesets so had to disable them temporarily to see how much my CF setup can handle :smiley:

If you’re doing any type of CF testing, CF Pro or higher plan would be useful as the Web, Cache and Firewall analytics and info available is much better for troubleshooting.

But @cscharff is correct 524 is an entirely different matter origin side from CF 429 errors from edge. Once your origin stops responding to requests in a timely manner, CF edge trying to contact your origin will also throw a lot of errors.

I don’t use Jmeter myself so not sure if that is a factor. I’m currently using k6 benchmarking tool i.e. k6-benchmarking/bench-ramping-vus.sh-3.md at master · centminmod/k6-benchmarking · GitHub :smiley:

EDIT: also https://support.cloudflare.com/hc/en-us/articles/115001635128-Configuring-Cloudflare-Rate-Limiting

HTTP 429 includes 429 responses returned from the origin if the origin web server also applies its own rate limiting.

Maybe this applies to your situation as you mentioned you are also doing rate limiting on origin Nginx server side?

8

I have made several tests so far but, cannot still see anything during the time ranges in the Analytics regarding blocking… And in the case that I was being rate limited by Cloudflare, responses should have the Retry-After header as per the Rate Limiting documentation, which I’m not getting.

The strange thing is that I have more than enough Rate Limiting settings in my server, and my server logs show only 200 s as a response. So I’m pretty sure I’m not applying any Rate Limiting on my side.

9

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.