Occasional 526 errors

[Bhooks]

I have a PageRule redirecting my image requests to a CDN (“Backblaze”). This mostly works fine, but every now and then I get a 526 error:

Domain: bhooks.com (you can just click on the search top right and scroll down, eventually you’ll get one.)

I’ve used the troubleshooting, Community Search, diagnostic center, google, you name it. But this 526 doesn’t make sense to me. Why would someone get a redirect loop every now and then?

The two pagerules there:

Some random info:
a) Full (strict) with let’s encrypt certificate
b) Image requests are proxied but website requests are not
c) backend: nginx kestrel asp.net

I’ve ignored the problem for a while, but it’s been persistent.
Thanks for your ideas!

Kind regards,
Roman

[sdayman]

526 indicates an SSL error. Did you see something that indicates a redirect loop?

Even an intermittent 526 doesn’t make sense. But at this point, everything from that subdomain is showing a 526.

I also have to point out that you’re violating Cloudflare Terms of Service 2.8 by only using Cloudflare as a media CDN.

Self-Serve Subscription Agreement | Cloudflare

[Bhooks]

Thanks for your reply, much appreciated!

Hmm well, I don’t know where to look really from this point on. Because e.g. from the following details I only read “error comes from cloudflare”, otherwise I don’t recognize anything as “suspicious”.

What do you mean with “everything from that subdomain is showing a 526”? Does every image loaded result in a 526 for you? Because for me it’s maybe 1 in 30.

Oh I’m shocked to hear this. I guess you’re referring to this part?
“Use of the Service for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as a Paid Service.”
But I’m wondering: The CDN is Backblaze. The blog is proxied, but the main website is not. I can’t proxy the main website because I have a mailserver running on there too. Sooo if I disable the CDN for the main website (i.e. 99% of media would come from Backblaze directly), would that be ok again?

Thanks!

[sdayman]

Go figure…now I don’t get any 526 errors. But it looks like you’re BYPASSing caching for the images subdomain now. (p.s. bypassing cache isn’t a viable workaround for ToS 2.8, as that data is still traversing the Cloudflare network).

@Bhooks

would that be ok again?

That’s up to Cloudflare’s Trust & Safety group. Most sites won’t even show up as a blip on their traffic radar and they won’t even notice.

As for the mail issue, most hosts assign a ‘mail’ hostname for that purpose, even if it’s on the same server. Then you can proxy the main site as well.

[Bhooks]

An right now I get tons: 1 in 2 returns 526. Seemingly random.
I haven’t changed anything yet. I didn’t turn off CDN or anything, that was just a suggestion.
I’ve also checked some backend logs: There is nothing in the letsencrypt log, and I found nothing suspicious in the nginx log.
Do you have an idea what I should be looking for and where?

Well right now Cloudflare’s monthly statistic says it serves 3GB/month from its cache. If I disable CDN that would go down to a few dozen MBs per month I’m sure. I guess that’s “under the radar”

Oh, as in “mail.domain.com” instead of “domain.com”? How can I use my @domain.com mail addresses then? Hmm do you happen to know an article detailing this concept? (I think I’m searching the wrong keywords).
Then I’d fix the mail issue and proxy the entire website again, fixing the TOS violation.

[sdayman]

You can’t hide under the radar with a cache bypass. But if it’s just 3GB in 30 days, they won’t even find that with a microscope.

For mail, if you have an MX record for example.com that points to mail.example.com, that tells messages that email for [email protected] should be sent to mail.example.com. That’s how mail servers work, as they don’t demand to reside on the main domain’s server. Then all email transactions for @example.com go through mail.example.com: Your mail app will point here, as well all external mail delivery systems.

[Bhooks]

Sidenote: Enabling or disabling cache on the PageRules changes nothing:

I still get the occasional 526 errors.

[user12010]

Getting the same thing here. The domain is frequens.com.

I think that the fact that these errors are seemingly random confirms that this is a Cloudflare issue, especially considering the corresponding Troubleshooting guide: https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#526error

In my case, I am not using any other CDN behind Cloudflare, just Heroku serving the files.

The error is so random that upon website load, out of 40 images 1 or 2 will have that error, but if I reload, they will be fine.

This is extremely problematic.

Can someone with the necessary privileges escalate this as an issue?

[Bhooks]

Any news on this? The issue is not subsiding…

[user12010]

I have been able to narrow down the issue:

This used to happen because I had disabled caching entirely, through a Page Rule.

Similarly, even after enabling caching, if I purged the cache and reloaded my website in my browser, it would show the same behaviour: one or two elements, most of the time images but sometimes /index.html as well, would fail with a 526 error code.

Thus I think this is a Cloudflare bug, which happens when Cloudflare needs to fetch many resources from the origin server at the same time, because they are not in the cache. In such a case, Cloudflare seems to randomly fail the SSL certificate verification.

It is pretty easy to replicate: purge the cache, reload the page, see 526 errors.

A temporary workaround to this issue, for end users like us is to reenable caching, and if needing to purge the cache, to reload as many resources ourselves after purging it, so that errors fall on ourselves instead of on our website visitors.

[Bhooks]

Unfortunately, that doesn’t help in my case.
Thanks for the update though!

[Bhooks]

On another note, I seem to get fewer 526 errors in development mode. But they are not gone.

[user12010]

Yeah, and actually thinking back to my workaround, it is actually really not a solution, just a workaround, because non-cachable items such as HTML pages or dynamic content run the same risk of seeing a 526.

This is a real heavy bug by Cloudflare.

[mitchell3]

This is also happening for me. Some people are seeing intermittent 526 errors. It’s just one every now and then, during regular use of the app - not high frequency. I’m also using Heroku. It started happening within the last few weeks tops, though its hard to pinpoint exactly when it started.

[user12010]

I have reached out to Cloudflare support, and they are being absolutely terrible. They still haven’t understood that this is a serious Cloudflare bug.

Did you try switching to Full SSL, instead of Full (Strict)? I wonder whether that would solve the issue, but I am not even sure.

[Bhooks]

I can’t seem to reproduce the issue right now. I’ll try again in 1-2 days.

I’ve tried Full (without Strict) before, it changed nothing.

[Bhooks]

It seems like the issue is fixed now.
If it’s also fixed for you @user12010 and @mitchell3, I’ll close the issue.

[user12010]

What makes you think this is fixed? Did Cloudflare notify you of a bug fix?

[Bhooks]

No, I simply haven’t had the error anymore in over a week. Do you still have the error?