Intermittent and random - Failed to load resource: net::ERR_QUIC_PROTOCOL_ERROR

What is the name of the domain?

example.com

What is the error number?

ERR_QUIC_PROTOCOL_ERROR

What is the error message?

Failed to load resource: net::ERR_QUIC_PROTOCOL_ERROR

What is the issue you’re encountering

Since last Thursday 20/06/2024 there has been a ERR_QUIC_PROTOCOL_ERROR happening intermittently and randomly on 5 of my AWS EC2 servers routed through Cloudflare. All 5 servers have the same setup with different branding. There is no pattern to when the error occurs. Once every so often (maybe 10 loads) this error will occur in console and the page will be blank. A reload may fix it, or a reload may lead to the blank page again. After a while reloading the page will work again. We have many users in different countries all experiencing the same problem. Notably, the users in Pakistan will get the error/blank page 100% of the time and resolve the issue by using a VPN to another country. I have made no changes to my servers or code in the past 2 weeks and we have never experienced this issue in the 4 years prior. It does not happen at the same time for all users. 1 user can load a page just fine and another can have the error. Disable the Google chrome flag Experimental QUIC protocol fixes the issue for some users. However when I have set this setting I sometimes get a ERR_HTTP2_PROTOCOL_ERROR in it’s place. Disabling the flag is not a solution for us because we have 1000s of users all over the world from different companies, so having everybody change a Chrome setting is not feasible. If I disable HTTP/3 (with QUIC) in Cloudflare it resolves the ERR_QUIC issue but again the new ERR_HTTP2_PROTOCOL_ERROR appears in its place. This problem is affecting all users regardless of location and browser

What steps have you taken to resolve the issue?

Increased ec2 server size (ram/cpu)
Increased ec2 storage (30% disk utilisation now)
Disabled HTTP/3 (with QUIC) in Cloudflare
Unproxied the domain (dns only) in cloudflare and tested without cloudflare (errors disappear - so cloudflare problem?)
Tried other non chrome browsers

What are the steps to reproduce the issue?

The servers are private so I won’t be able to share

I believe I have just confirmed it’s an issue on Cloudflare’s side rather than the server or browser.

When I got the ERR_QUIC_PROTOCOL_ERROR (now ERR_HTTP2_PROTOCOL_ERROR because I turned off HTTP/3 (with QUIC) in Cloudflare) I reloaded the page a few times to confirm the error was sticking around. I then changed my Windows host file to point the domain directly at the EC2 instance public IP (thus connecting to the server directly and skipping cloudflare) reloaded the page and it worked as normal. I then reverted my host file so that when I access the page it goes through cloudflare again and still the ERR_HTTP2_PROTOCOL_ERROR again (confirmed through headers that CF variables appear).

After about 10mins I reloaded the page and it was working as normal again. Which matches the intermittent problem.

I’ve read on other posts that Chrome doesn’t like malformed headers. It’s possible that cloudflare is supplying malformed headers randomly

We have the exact same error since the 20.06 too !

Disabling Quic in Chrome, get us also the same error ERR_HTTP2_PROTOCOL_ERROR

In our case, it looks like related to HTML document size. When it’s too big, the html page is partly loaded (around 800KB). On smaller size even 100KB it is still working.

It was working without any issue before 20/06

If we bypass Cloudflare the same pages are working properly

I found the source of the problem, turning off the ZSTD compression in Chrome fixes the issue.

Go to chrome://flags/, search for Zstd Content-Encoding, and disable it.

It looks like the support ZSTD is quite recent,

Thanks. I did notice that when I am getting the problem the compression was “zstd” and when the page is working the compression is “br”

Chrome flags is not a solution for us as we have like 3000 users all over the world with different levels of knowledge, and increasing every day.

I will test the flags solution personally.

I have paid for pro cloudflare to submit a ticket so will keep you updated @Lionel1

It’s for sure not a solution, but just a workaround.

It’s quite surprising we can’t disable this compression at Cloudflare level. At least I didn’t find any option in the admin interface,

Hoping Cloudflare will be reactive.

cc: @mbullock - zstd related

Could someone possibly capture a Netlog of the issue and email it to me walshy@?

Sent.

Just an update. My pakistan guy who would get the errors 100% of the time unless on VPN. I confirmed he was getting zstd header compression. After changing the flag it has set to “br” and works flawlessly on his pakistan internet.

Thanks, I’ll check it out and raise it to the relevant team!

1 Like

In the meantime,
@Walshy is there a way to disable ZSTD compression at Cloudflare level easily in the admin interface ?

Can confirm we have also been seeing this since Thursday too. If you need any info from our side let me know.

It looks like, it’s only possible in Enterprise version, by creating a Rule.

So let’s try to be patient :slight_smile:

Hi. We are having the same problem.

It seems related to some operations within the workers. However, everything seems ok at the moment.

Is anyone still experiencing the problem? Have you found any solutions?

Hi Everyone,

Thank you for your patience and apologies for the issues you encountered. We rolled back the update to the compression module on June 24, 2024. The issue should now be resolved. Please let me know if you are still experiencing any issues.

The change incorporated a new compression algorithm, Zstandard. Zstandard is a fast compression algorithm that provides high compression ratios, offering significantly better compression performance and efficiency compared to Brotli and Gzip.

We appreciate the reports you provided and will investigate the root cause. We will keep you informed of our findings.

Best regards,
Matt

2 Likes

Since I encountered this bug, I learned about ZSTD from Facebook.
And played with it on my Linux, waah it’s fast ! :zap:

On pages where ZSTD compression was working, I saw that there was a 20% gain over Brotli on html, css and js :dark_sunglasses:

2 Likes

Yes it is exciting we are hoping to be able to offer this to you all without needing to change the Origin. The engineer is back and looking at the traces you sent through - appreciate the help.

Matt

1 Like

This topic was automatically closed after 15 days. New replies are no longer allowed.

Hi Everyone,

We identified an issue with how we parsed large files and converted to Zstd. We think we have a fix and will slowly roll this out to our Free plans over the coming week.

Any issues please ping me here / let me know!

Matt