Cloudflare: Error 524. Service Outage

[adeko]

Issue

We are using Google Cloud Platform VM instance with SSD (us-central1-f) and Cloudflare free plan.
Since April 15, 2020, we have started getting a lot of Error 524.

First, we noticed that social media could not get shared content OG meta data.
Later, we have gotten a lot of 5xx errors in Google Search Console with the list of de-indexed pages.

We have analyzed logs, the errors time was random, but we’ve noticed outage spikes the time content is shared on social media.

Therefore, we tested website performance after sharing the post, and received Error 524. We also tested website using sitemeer.com, see screenshots below.

Nginx

According to Cloudflare, Error 524 indicates that Cloudflare made a successful TCP connection to the origin web server, but the origin did not reply with an HTTP response before the connection timed out.

We have checked website logs, and noticed that we do receive request from Cloudflare, that it was processed, but Cloudflare refuses to receive response.

First we thought that website do not respond quickly enough, so we have optimized website local cache to send cached content after the first request.

However, it did not help.

Nginx Proxy Cache

The second thought was that nginx for some other reason cannot send response. So, we additionally installed nginx proxy in front of webserver, with cache enabled using ‘proxy_cache_background_update’ option. That allows starting a background subrequest to update an expired cache item, while a stale cached response is returned to the client. That improved website response even more.

Unfortunately, new tests showed that webserver works fine, and now timeout is at the nginx proxy server.

Traefik

To make sure it’s not Nginx problem, we decided to install some not-nginx proxy in front of current setup.
We have chosen Traefik. It is written on Go (Nginx uses C), fast and has built-in Let’s Encrypt wildcard certificates support.

Alas, new tests showed that webserver and nginx proxy work fine, and now timeout is at Traefik.

Conclusions

At last, on Friday, May 8, 2020, we have disabled Cloudflare Proxy. Soon, Traefik was able to receive Let’s Encrypt certificate via tlsChallenge.

Since then, website works flawless, no more connection and response issues.

Therefore, we suppose, this might be some Cloudflare bug, that does not allow webserver to send response.

P.S.

In addition, we noticed significant non-human traffic drop, maybe just coincidence.

Website: https://watchward.com/

[cs-cf]

A 524 error indicates that Cloudflare made a successful TCP connection to the origin web server, but the origin did not reply with an HTTP response before the connection timed out. Review the suggestions in this Community Tip for troubleshooting ideas.

[adeko]

The problem is that origin did reply with an HTTP response, but Cloudflare ignore it.

I’ve also installed nginx cache server between origin and Cloudflare, with proxy_cache_background_update. Origin response is OK, but now proxy cannot send response.

I’m configuring server to run without Cloudflare, then will run more tests.

[yolox]

I have same problem, last 5 days some many request getting timeout some reason.

[adeko]

We have not found any problems with our hosting, thus we disabled Cloudflare Proxy. No more issues since then.

I’ve updated first post in this topic regarding our issue.

[adeko]

I suppose error 524 is somehow related to CF proxy, thus CF certificates is not an option.

Cloudflare: Error 524. Service Outage

Issue We are using Google Cloud Platform VM instance with SSD (us-central1-f) and Cloudflare free plan. Since April 15, 2020, we have started getting a lot of Error 524. First, we noticed that social media could not get shared content OG meta data. Later, we have gotten a lot of 5xx errors in Google Search Console with the list of de-indexed pages. We have analyzed logs, the errors time was random, but we’ve noticed outage spikes the time content is shared on social media. Therefore, we te…

[sandro]

A 524 is when Cloudflare cannot reach your server. The certificate itself is not really involved here. But, if you still have the issue, we should continue that in your thread

[adeko]

Could you point me where it’s said that CF certificates can work without CF proxy.

Also, error 524 means that CF actually can reach your server but does not get a response.

[sandro]

I am not quite sure what you mean. A Cloudflare certificate will always be on the proxy, hence there is no need to discuss whether it works with or without.

[adeko]

Error 524 means CF proxy does not work, so CF certificate also won’t work.

[sandro]

Please see what I wrote earlier. The certificates are completely unrelated to that issue.

[adeko]

It’s related to your post, before you split the other topic.

@sandro

Additionally to what @adeko already suggested you can also check out Cloudflare’s Origin certificates. They are free as well, easier to maintain, and come with longer validitity periods. The downside is they are only valid in a proxied context and not trusted by browsers (only by the proxies).

[sandro]

That is still unrelated to your issue. That is simply yet another certificate which can be installed on the server.

[adeko]

Yes, you are correct. I was answering to your post in other topic, but it was redirected here because of split.

CF certificates are not related to the 524 error directly, because it happens between CF and origin server, where certificates are not applied.

But in general, it is related, because if you cannot use CF proxy, you cannot use CF certificate.

[sandro]

That is exactly what I addressed earlier, an Origin certificate only works in a proxied context.

[adeko]

And that is exactly what I addressed earlier, an Origin certificate only works in a proxied context. So if you have error 524, you cannot use CF proxy, thus CF certificates is not an option.

[sandro]

If you have a 524 error, you need to address that. Again, the certificate is unrelated.

[adeko]

I cannot agree completely.
Both CF certificates and CF error 524 are related to CF proxy.
But I understand what you are saying, that CF certificates are not directly related to the CF proxy bug.