Slow/Unusuable page loads with Chrome 124, HTTP3 / TLS 1.3 and perhaps comcast

Hi, We have been having an increasing number of reports of “slowness” over the past few weeks. Having users disable QUIC in chrome:://flags seemed to take the (on average) 60s+ stalls down to 30s. User reports after disabling HTTP/3 and TLS1.3 in cloudflare settings have all been positive with the slowness disappearing. One strange data-point is: all but 1 user has been in the UT/SLC region using comcast.

We are assuming this has something to do with kyber and https://tldr.fail, however the only settings in our control seem to be disable both HTTP/3 and TLS1.3. Disabling TLS1.3 did /not/ disable kyber from being used in QUIC and we still had slow reports until we disabled HTTP/3. AFAICT TLS1.3 (or HTTP/3) do not have a way to configure cipher suites, we seem to be a bit stuck with disabling these for now. :frowning:

At the very least, this seems very surprising, and we are at a loss as to what to do about it. Downgrade chrome? Report the bug to chromium? For now our best option seems to be disabling HTTP/3 and TLS1.3.

Postscript:
We had a few users run ping -D -c 300, traceroutes, and tcpdump. one tcpdump we looked at had a high number of tcp re-transmits (over 3000 in the span of a few minutes), which looked similar to some reports of kyber issues we found on reddit. QUIC remained a blackbox, not sure if that also was having trouble handshaking/re-transmitting (did not seem to get lucky and capture the handshake with chrome logging the ssl keys).

Ping and traceroute mostly looked clean (~1.6% packet loss) to the cloudflare host, we also had them try various MTU sizes which seemed to settle around 1478. Initially for one user it looked like there might be some strange/lossy hops over ipv6, having them switch to ipv4 did not solve anything.

As with all good network issues, we are not positive if it’s kyber, QUIC, TLS1.3, or some bad hop in say comcast… do they have some tls intercepting broken middleware? Maybe by default the modems have some kind of broken WAF enabled? Or perhaps somewhere on the SLC cloudflare side…?

1 Like

We are seeing the exact same thing. We had to go so far as to static route traffic to the three IP addresses that are hosting our Cloudflare proxied sites to our CenturyLink connection. We opened a ticket with Comcast and they are, so far, not willing to take the report very seriously.

We are experiencing the same issue.

Disabling HTTP/3 seems to have solved our major performance issues. I think, however, that the problem is two fold. I think there is packet loss between Comcast and Cloudflare. The two will need to get together to solve that part. We are consistently seeing 1-2% packet loss to our Cloudflare IPs via Comcast and zero packet loss via our CenturyLink connection. Until that can be resolved, we will have to pin our Cloudflare traffic to CenturyLink.

Disabling HTTP/3 solved our performance issues, too. Interesting.

Hello,

Similar issue here with Comcast and today I had another customer from a different ASN having the same issue, TWC-20001-PACWEST

HTTP/3 is disabled, it solved most of the issues but there’s still some users with persistent issues unfortunately. Hopefully, the support can help.

You can ask the people having issues to send you traceroute logs and your-domain.com/cdn-cgi/trace and forward it to the support as well.

A few more interesting details (I work with @alex.hunsaker ) - one thing that led us down the HTTP3 path is that I was experiencing moderate slowness and turning http3 support off in my browser (Chrome) made our home page load go from 10s → 2s, very reliably and reproducibly. I’m on Google Fiber, not Comcast, and was unable to reproduce any of the packet loss that we saw on Comcast.

We also saw weird behavior in our HAR exports:

I’d see batches of requests all complete at the same time, and requests for tiny static files (headers indicated they were cached by Cloudflare) taking as long as an API call that had to go all the way through to our services. Disabling http3 made this go away as well.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.