Troubleshooting 520 for Requests from Specific IP

[ddosdex]

I’ve gone to “WAF > Tools” and added an IP address and selected Allow. I don’t want any request from this IP address to be blocked by Cloudflare for any reason.

But even after waiting 5 minutes, I still get 520s from Cloudflare when I make a cURL request from this IP. The same cURL requests works from my local machine (and several other machines) so I’m confident my origin server is healthy.

Here’s the cURL response:

< HTTP/2 520 
< date: Fri, 29 Sep 2023 01:46:53 GMT
< content-length: 0
< cache-control: no-store, no-cache
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=N83kK5HJNn6f42bXZwSIwPQZ3zICO0b%2BG8W%2BNMI9J%2BmS3WOMMv8LWAqwI9xa7wkNVP%2F0H09SWnwl9oqXnBhnIyX206%2F7uIFdrbv1aQsDVotbiTcyTqSqKzSazowT8XCqB7GGaw%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 80e08df3cdc03943-IAD

And here’s screenshot of “WAF > Tools”:

I don’t see the IP being blocked in my WAF activity log.

I’ve purged the cache for good measure.

I’ve also hit my origin server directly from this IP (bypassing CF) and I get the response I expect.

But I still keep getting 520s when making a request from this specific IP address via CF, when a bunch of other IP addresses work without problems.

How can I go about troubleshooting this?

[cbrandt]

Hi,

A 520 reflects some difficulty between the Cloudflare edge and your origin server, at which point WAF has already allowed the request. You’d get a 403 if WAF blocked it.

Troubleshooting a 520 is rather challenging, because origin servers may have totally different configurations. You could this community for topics related to 520, and use the search advanced feature to filter only resolved topics. This might lead you to cases where disabling HTTP/2 to origin solved the problem, or disabling HTTP/3, or changing some server configuration. I’ve seen someone claim it was a WordPress plugin that was changing the URL in a way that the server wouldn’t understand etc. etc.

You can use the new Trace (beta) feature, available at the Dashboard > Account page, to trace internally your request and see if that gives you an idea whether the specific request is triggering any rules that may be playing a role here.

Also, you can try the Dashboard Analytics to see if it returns the origin status code that is associated with the edge status code 520.

Community Tip - Fixing Error 520: Web server is returning an unknown error

Error Try the suggestions in this Community Tip to help you fix “Error 520: Web server is returning an unknown error”. Background Error 520 is essentially a catch-all response when something unexpected happens or when the origin server incorrectly interprets or does not tolerate a request due to a protocol violation or an empty response. A 520 error occurs when the connection started on the origin web server, but that the request was not completed. The most common reason why this would occur …

Please see if you can edit the title of your post, so that it properly reflects your search for 520 troubleshooting, so that it gets the attention of the right set of eyes. (If you can’t, flag your own post with the “other reason” and ask mods to change the title for you.)

[ddosdex]

Updated title to reflect 520.

A followup question:

If I get a 520, can I be certain CF attempted to hit my origin for this request? Or could there be any reason that CF didn’t even attempt to hit my origin server? Like for example if CF sees too many requests from an IP getting dropped by my origin server in a short span of time, could CF just “give up” on serving requests from this IP until some time has passed (and return 520s without hitting origin)?

The reason I ask is because I saw a spike in 520s last night. I dug into my analytics and noticed a very high variance between IPs and how many 520s they received. In other words, requests from some IPs were nearly always getting 200s and requests from some IPs were nearly always getting 520s. I don’t have any IP specific logic in my origin server afaik. Eventually the problematic IPs started getting 200s again without any change from my side.

If we operate under the assumption that my origin server was temporarily overloaded and CF was attempting to hit origin for every request, I’d expect 520 responses to be spread uniformly across all IPs. But this doesn’t appear to be the case.

Any thoughts on this?

[cbrandt]

@ddosdex

Updated title to reflect 520

Thanks.

@ddosdex

requests from some IPs

Are you certain that requests coming from the IP that gets 520 do not also contain some element that might potentially trigger the error? Such as HTTP version, compression, or whatever other factor the origin server may have an issue with?

I’m just speculating here, as I’m far from being qualified to discuss server issues. I’m just raising the idea that perhaps you should investigate request details other than IP address per se.

Let’s hope someone with server admin expertise join in and give your more specific suggestions.

[ddosdex]

I’m pretty sure, because my hack fix was to change the problematic IP address to another Elastic IP (from AWS) and it started seeing 200s immediately.

I’m now trying to figure out the root cause because I’m afraid the same thing might happen with my new IP.

[ddosdex]

As you rightly pointed out, it was a problem with my infrastructure upstream. Specifically, an issue with my load balancer and how it was routing traffic. I don’t think the details are relevant here, but bottom line: nothing wrong with CF.