Sporadic corrupt(?) output from functions route

I’ve added two routes to the functions directory that act as proxies for mutation of subdomains to sub-directories.

For most part, I referred these articles and wrote a variation of what was proposed:

These proxied origins are:
docs.lyla.ai (docusaurus @ separate cloudflare pages application)
blog.lyla.ai (ghost @ aws ec2)

All this was working absolutely fine till approx. 3 days ago when I updated all builds (the proxy code was untouched) and now, not always (how I wish it was so), but intermittently, instead of a proxied html page, the browser shows what looks like binary data.

I’ve attached a screenshot capturing this output and the associated network request. I’ve compared the headers of a failed network request with a successful one, and I couldn’t spot a difference.

Please help!

Hey,

Can you try to disable Brotli and see if it resolves the issue?
You can disable it here: https://dash.cloudflare.com?to=/:account/:zone/speed/optimization/content > “Brotli”

2 Likes

I really appreciate that you responded. Thanks!

I’ve been trying to figure out how to reproduce this issue consistently, or at least more frequently, so I can test possible solutions. I think it’s got something to do with the combination of Brotli and coldstarts.

This site is not live yet so I’m the only one generating any traffic, and the function routes are not at domain root, so they are prone to coldstarts (Eliminating cold starts with Cloudflare Workers).


If Brotli is turned ON and a VPN service is used to access the proxied subdomain routes from different geolocations, this error is much more frequent, and:

  • When I do get the error, it’s persistent
  • This issue can manifest for a proxied HTML document or an image/CSS/JS asset referred therein
  • It doesn’t matter if I force clear browser cache / hard reload page. It keeps failing for a few minutes
  • After a while (from the same VPN geolocation), it doesn’t fail till I’m done for the day
  • If I switch to a different VPN geolocation (to possibly engage a different infrastructure segment), the issue shows up again
  • I’ve tried switching back and forth between two VPN geolocations, one with issue and one without; the behavior keeps holding the same pattern. One geolocation keeps failing while the other one keeps working fine

If I switch Brotli OFF, and repeat these tests, I don’t face this issue.


It’s possible I’ve come to the wrong conclusion based on my observations. But if I’m right:

  • Is this a bug in Cloudflare Pages/Workers when using Brotli?

  • Is it possible for coldstarts to manifest just as delays and not failures?

  • What’s the duration after which a pages function/worker is moved to a cold state?


If it helps, here’s the HAR export of one of the failed requests:

Amazing thanks for the info!
Where did your assumption about cold starts causing the issue come from? They will just cause some delay, from what you described you hit a hot site and had the issue there.

Yes it seems so. I’m working with the relevant team on it :slight_smile:

They should already.

That’s variable. There’s no constant time. If you have just a few requests a day to the same Colo though, it’ll likely always be hot. It can take a long time to go cold.

A (bad) guess then; if not an assumption :face_with_diagonal_mouth:

Because even if this issue is sporadic, I’m trying to see if there’s a pattern.

For the past hour, I’ve been trying to repeat the same test as yesterday. Switching my geolocation via VPN and accessing the proxied subdomains.

  • Most geolocations are working fine
  • UK was stuck in a erroneous state for ~10 min but it’s resolved and working
  • Aukland, NZ has been stuck in the same erroneous state for longer and, as of now, still not working

I can’t guarantee if by the time you see this post and try these URLs, if the issue would’ve been resolved. But if you access these URLs from a client location in Auckland, NZ:

You might see an active error state.

I’ve removed Cloudflare Access from the main domain. The proxied subdomains are still protected by an Access policy though.

The fact that the error state persists is strange. As per docs, Cloudflare doesn’t cache HTML documents unless explicitly configured to do so and I haven’t done that.

This implies the worker route is triggered on every hit and if it fails, it keeps failing for an indefinite period.

Here’s another possible clue, this was not happening a week ago. I could be off by a few days. But it’s a recent development.

I’m not going to keep Brotli turned on for much longer. A few hours at most.

If you need any information from me to help reproduce this issue, please let me know.

If the concerned team finds and fixes this bug, will this topic be updated? If it’s not too much trouble, I’d like to learn what the bug was :slight_smile:

Thanks again!

Yeah the team has repro’d and are investigating. You’re good to disable Brotli and we’ll update here when it’s fixed :slight_smile:

1 Like