429 Too Many Requests when using CURL or Proxy

When doing a request to a site behind Cloudflare using CURL, I always get error 429.
How do I get passed this? What is Cloudflare checking? I know it’s not because of “too many requests”, because I get it even if I make ONE request after hours of nothing.

I can make the request in a Chrome without any problems, but if I save it as a CURL-request in the dev-tools of Chrome and try to run it, I get Error 429.

So same request, same header/body etc. Is Cloudflare checking things like TLS versions and block on those? I cannot be the only person who stumbled upon this. Thanks.

Btw, I see the same issue if I use Fiddler to analyze the request, so it must be something that Cloudflare is detecting and returning that 429 error.

Greetings,

Thank you for asking.

May I ask if you are trying to do a cURL request to your domain/Website behind Cloudflare, or rather to some other domain/Website which is using Cloudflare? :thinking:

The former. Someone else website. It’s obvious that Cloudflare is detecting something and blocks the request. Even if I send the EXACT same headers with CURL, it still gets blocked. Through browser still works fine.

Btw, I would be happy to send you the URL if you want to try it yourself.

Thank you for feedback.

You might also be hitting some Rate Limiting Rules which owner of that particular Website has configured with specific parameters (time-frame, same IP, frequency, requests …):

I am afraid you’d have to contact the Website owner to check this for you and allow your IP to pass by them, if that’s a legal HTTP request from you or your origin host to their Website.

Despite cURL might be blocked manually/individually by the website owner on his origin host, and even on Cloudflare using a custom Firewall Rule or User-agent blocking pattern or as already mentioned, a Rate Limiting method.

Well, that’s why I am asking what kind of rules Cloudflare can trigger on. If I send the exact same request, including the exact same headers etc, what kind of differentiators could there be? Only thing I can think if is TLS versions or cryptos.

Like I said, I can take that CURL command and run it from a completely different location, different country, different IP and first ever request made and it still triggers 429.

If I am sending the exact same headers, they simply can not block on User-Agent etc, as it would be exactly the same as the browser, where it works fine.

Using Firewall Rules, we can block cURL because the HTTP request has got “curl” in a user-agent string with a version by default.

You could try adding some different user-agent with parameters like -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0' while sending a cURL and see if anything different at all.

But, even with that, I don’t guarantuee beacause the Website owner could have enabled Browser Integrity Check or some other Security options available to him, which would throw you some other or similar error.

You can even try, if you know, the origin host IP and connecting directly to it by using --connect-to to use the website server’s IP address.

Again, I don’t guarantuee you a success, because some other security measurement which owner of that particular Website has enabled and is using, could restrict the request and challenge/block it further.

Even Bot Fight Mode could block the request:

There are a lot more which could be “in the middle” due to the available security options to the Website owner to protect his Website:

The best case scenario would be to contact website owner to whitelist and allow your IP.

1 Like

Thank you for trying to help, but I think you missed one important part. I am sending the EXACT SAME request as the browser. That means, same headers, same body, same exact request. That means things like user-agent and other headers are already the same. I am not using CURLs default values. I am sending the exact same request. Even verified by running the request through a proxy to analyze the request. It’s easy to generate a CURL request in Chrome by using the developer mode and “copy as cURL (bash)”. It gives you the full command including all headers etc.

However, this “Browser Integrity Check” sounds interesting. How does Cloudflare check that? These are the things I am interested in. I just don’t see how Cloudflare could know who is making the request, when the request looks exactly the same as a browser.

I checked this “Browser Integrity Check” and it looks like it just checks for bad/missing headers. That would not be true in my case as I am sending the exact same headers as Chrome. For all intents and purposes, Cloudflare should think the request is coming from a Chrome browser. Unless it also checks for things like TLS support etc. One think I could think of is that it matches Chrome version send in the user-agent header with a known list of supported TLS cryptos and if the TLS handshake presents cryptos that does not match, then trigger some flag that then the website owner could put a Rate Limit to 0 on.

Customers using Advanced Rate Limiting can use dozens of different values to decide if they should apply rate limiting to your request: https://developers.cloudflare.com/ruleset-engine/rules-language/fields/

It’s just speculation trying to guess which one it is - only the site owner will know.

Bot Management customers can use JA3 fingerprints to tell if your client is actually what it’s saying it is GitHub - salesforce/ja3: JA3 is a standard for creating SSL client fingerprints in an easy to produce and shareable way. - which is what you’re mentioning in regards to the TLS negotiation

2 Likes

Ah! Now we are getting somewhere! Thank you very much for this.
Looking at the list of fields available for Rate Limiting, most (if not all) goes away because my request sends the same request. Only the JA3 fingerprint would probably differ in my case, so I would just need to match a valid (and accepted) user-agent with a genuine cipher list. Awesome! Thanks!

I just wanted to thank you again! Using a library to spoof the JA3 fingerprint solved the problem as I suspected. Now I am able to bypass those useless 429 errors. Blocking on something that the client can control is pretty stupid in my opinion.

It could be JA3 or it could be something else that JA3 uses which they blocked on - it’s impossible to tell without seeing their ruleset.

JA3 is helpful for let’s say, you’re getting attacked from various IPs/User-Agents/etc but they have a common fingerprint (i.e the same HTTP client). Much like the rest of the WAF or anti-bot suite, it’s just a single layer.

There’s manual firewall rules, rate limiting, managed rulesets, OWASP rulesets, Bot Management/Bot Fight Mode which goes off machine learning/heuristics/IP reputation, threat score, yada, blah, blah

Anyone persistent enough to just fetch your data will get past them since that’s not really a threat - but any vulnerabilities like Log4J / SQLi get caught by the WAF and brute-force attacks like L3/L4 or HTTP DDoS gets caught by the usual DDoS protection + rate limiting.

I don’t really know why they have a rule to block stuff like cURL but, y’know, they gonna do what they gonna do

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.