522 Response with Brotli Enabled

See ticket #2236828 for issue.

New thread at the request of @simon

No, we are not hosted by GoDaddy. We are “self hosted”.

FWIW… we experienced this EXACT issue (randomly over a 24 hour period, 50% of all transaction were getting 520 errors. The solution we found was to disable BROTLI compression. We never did get any useful information from CloudFlare what the real problem was, but we now operate without BROTLI enabled and have not had an issue since. The Ticket Number was #2236828.

1 Like

That’s why choosing a good host is important :grin:

Do you disable Brotli on Cloudflare dashboard or do you disable in your host?

In the CF Dashboard. We use GZip in our load balancer.

As a follow up, we had been using Brotli for about 4 years and never had this kind of issue. Because of the importance of SSL Acceleration, we did not have the option of disabling CloudFlare without drastically increasing the latency to our customers across the world. In addition, the security risk of exposing our Origin IPs to the “wild wild west” was an unacceptable risk.

Simon, the ticket number is #2236828. Our issue started 19 days ago and was ended only after Disabling Brotli in CloudFlare.

We disable Brotli about 12 days ago and the issue stopped.

99% of our customers use a Brotli compatible browsers (Safari, Firefox, Chrome, Edge, Opera) and about 90% of those are on the latest releases.

Even CURL through the CF Edge was getting 520’s. Health Checks in CF were getting 520’s.

To test, we set up a simple URL “ping” to check via both CF Edge and Direct to Origin. With Brotli active, 50% of the transactions FAILED when going through the CF Edge. Zero failed when going directly to the origin server’s external IP addresses (firewall, load balancer, virtual server).

After disabling Brotli, our transaction rate went from about 50% 520 response to 0% failures.

I suppose it is possible we were seeing something that was being rolled out across the CloudFlare edge servers, but even that seems unlikely. We have customers across the globe and every location (CF Edge) was getting similar issues uniformly across every edge location.

Unfortunately, I cannot even consider enabling Brotli since the impact to our site is catastrophic.

Because our origin servers block all requests that do not come from CF Edge servers, direct access would not be possible (that would get a 520… by design, PCI compliance and CF recommendations).

I am just disappointed that when we collected the information requested… we got absolutely no assistance from CF support. It was only after figuring out that it was NOT our origin server’s issues that we started disabling CF functions to see what the impact was.

Brotli was/is the “smoking gun” as far as we have been able to determine.

As to why Brotli being active would result in a 520 response, I would have no clue. I agree it seems odd, but it is what it is.

FWIW, when I read the post about Brotlii disabled it via CF, Speed, Optimisation for the domain - but the 529 issue persists.

@user7472 I don’t think your issue is resolved (I still see 520s for one of your domains). Are you using GoDaddy for your origins? If so you might be impacted by this. If not, let’s start up a separate thread where we can look at your issue separately.

Simon, new thread started for this issue.

here is the link:

Hi there - your ticket looks like it was about 522 errors - which are network timeouts - they’re not related to 520 errors.

Neither of these errors correlate with brotli content compression - since brotli is applied between the visitor and Cloudflare - and 522 / 520 errors are experienced between Cloudflare and the origin.

So - my recommendation would be to re-enable Brotli and monitor for errors. Happy to take a look if you can reproduce an error after that.

@simon,

I do see that there have been 480 “520” responses in the last 72 hours (based on information from analytics) and ALL appear to originate from the Madrid, Spain (MAD) Cloud Flare edge data center.

You are correct. Our issue was with 522 errors.

Jim

Hi Jim - your 522 errors are TCP connection timeouts between Cloudflare & the origin - typically that will be a network issue at the hosting provider or them rate limiting our IPs.

At this point - I would recommend re-enabling brotli if you want to and then taking a fresh look. As I have explained - there’s no real correlation between eyeball compression (brotli) and origin failures (520 and 522 errors).

I would normally agree with you and given that we contacted our upstream provider (Cogent Co) to address the issue FIRST. After a number of back and forths with them, we also determined that the issue was most likely somewhere between CloudFlare and Cogent Interconnects, but I could not get Cogent to comprehend what the issue was.

After almost a week of pushing on Cogent and CloudFlare, we decided to start disabling “features” in CloudFlare that were “Nice to have, but not required”. Brotli was the one that almost immediately resolved the issue.

I can give it a try enabling it, but all our testing “direct to origin” server did not produce failures. Only traffic from CF Edge to Origin seemed to have an issue.

Simon,

I have re-enabled Brotli. Watching to see what happens

The issue was sort of random… It would go for a couple of hours with no issue, then start about 50% failure for a couple of hours, then stop.

That went on for almost a week.