524 Origin Timeout errors via Cloudflare's AKL data center

[user7291]

What is the name of the domain?

What is the error number?

524

What is the error message?

524 Origin Timeout

What is the issue you’re encountering

524 Origin Timeout on some requests routed via AKL

What steps have you taken to resolve the issue?

For the past few weeks I’ve been observing 524 Origin Timeouts in my Cloudflare logs (and receiving reports from customers about timeouts and slow performance).

However, this is only occurring on a small subset of requests which are routed via Cloudflare’s Auckland, New Zealand (AKL) data center.

For context, the origin server is located in Seattle, Washington.

1. There are no long running processes on the origin server
2. The origin server is not overloaded in terms of CPU usage, server load, or traffic
3. The origin server has plenty of available free memory
4. There are no origin server responses taking anywhere close to 100 seconds. Logging of the execution time of all http requests confirm that every response is taking less than 1 second.
5. All cloudflare IPs are allowlisted on the origin server
6. I am aware of, and have ruled out everything mentioned in Community Tip - Fixing Error 524: A timeout occurred

At the times indicated in my Cloudflare dashboard that these 524 Origin Timeouts are occurring, there are no corresponding entries in origin server access logs of these incoming connections - i.e. they don’t actually appear to be reaching the origin server.

Over the past 24 hours for example there have been a total of 62.26k requests to this origin, including 47.29k from the US, 7.85k from Canada, 7.42k from Australia, and 2.51k from New Zealand.

Out of those 2.51k requests originating from New Zealand - all of which were routed via AKL - 51 resulted in 524 origin timeouts (i.e. approx 2% of requests).

I am not observing any 524 Origin Timeouts for any of the other 63.75k requests that are routed to the same origin via other Cloudflare Data Centers.

It’s been a similar patten over the past 7 days - out of 17.22k requests routed via AKL during this period, 329 have resulted in 524 errors (approx 1.91%). By comparison, over the same 7 day period, 304.72k requests routed via other Cloudflare Data Centers to the same origin have resulted in zero 524 Origin Timeouts.

I am on a Cloudflare Pro plan, and raised a ticket (01412932) on this issue on 4th March (22 days ago) - I have yet to receive a response.

There is no indication on the cloudflare status site of any issues with AKL, however, having spent the past three weeks with my hosting provider ruling out various potential causes at the origin, we are at a loss - so I’m reaching out here in the hope that someone may be able to offer some insight (or someone from Cloudflare may read this and offer a response, given that my ticket appears to have so far been ignored)

Screenshot of the error

[user7291]

So I’ve performed more testing…

• Test 1: Grey Cloud (from NZ): Response times 586ms - 631ms
• Test 2: Orange Cloud (routed via AMS): Response times 347ms - 834ms
• Test 3: Orange Cloud (routed via AKL): Response times 686ms - 2.7 mins

…the latter of these tests results in 524 errors, and also the odd 499 “Client Closed Request” error.

As previously mentioned, there is no record in the origin server’s access logs of those connections which time out having actually reached the origin. Nor is there any indication in the server’s access logs of the execution time of any request which did make it to the origin taking longer than 1 sec to execute.

In Chrome Dev Tools, some of the requests routed via AKL are showing “stalled” for exactly 1 minute (see attached screenshot). This does not occur if the same URL is accessed yet routed via a different CF data center. From my understanding, ‘stalled’ states can be caused by proxys (i.e. Cloudflare)

Other things I’ve ruled out:

• Completely disabled CSF and mod_sec on origin - no difference
• Tried with Keep Alive both enabled and disabled on origin - no difference
• ‘allowlisted’ my test NZ IP (used in Tests 1+2 above) on cloudflare (to bypass any CF security/WAF configuration rules) - no difference

I am therefore confident that the cause of these poor response times and 499/524 errors is not an issue with my origin server, nor with my particular CF settings - but rather an issue, either:

1. With CF’s AKL data center itself, or
2. The routing/connection between CF’s AKL data center and my hosting provider’s data center (For reference, the hosting provider in question is @KnownHost , and the origin server is located in their Seattle, WA data center). This is also not the first time that there have been issues between CF and KH - so this is not without precedence.

Right now, my only option for customers in NZ is to “grey cloud” (bypass) CF completely, which isn’t a satisfactory or long-term solution, but it’s currently the only way to prevent these customers from experiencing poor performance and timeouts as a result of being routed to the origin via AKL.

For info: CF Case ID - 01412932 (Still unanswered after 23 days!) / KH Ticket ID - KH202503ZN3K3C