Fetch from origin if non-SSL request

[DanWeb]

Hi,

I began using Cloudflare about a month ago.

Before Cloudflare, my PHP script would handle any non-SSL requests by issuing a 301 redirect to the correct SSL-based URL.

I would like this behavior to continue since onboarding onto Cloudflare, but it is not working.

My SSL/TLS encryption mode is Full (strict). The origin server has its own EV SSL certificate.

When I began using Cloudflare, I came across the problem that if a non-SSL request is made when the “Always use HTTPS” option is disabled, a 307 redirect occurs to the SSL-based version of the URL. When the “Always use HTTPS” option is enabled, a 301 redirect occurs to the SSL-based version of the URL.

This is not the desired behavior because our URL structure has changed over the past 20 years that the website has been running. Most of the incoming links we have pointing to our site from 10+ years ago were before we switched over to HTTPS, and they are in our old site structure.

If our PHP script on the origin server detects a non-SSL request, it issues a 301 redirect to an SSL page that uses our new URL structure. Additionally, if our PHP script on the origin server detects a URL in the wrong structure, it issues a 301 redirect to the new URL structure.

Since using Cloudflare, HTTP->HTTPS URL rewriting seems to always happen on the edge. Therefore, two redirects are occurring one after the other: first HTTP to HTTPS, and then a second from our old URL structure to our new URL structure. This is both inefficient and is bad for SEO. The desired behavior is to allow my PHP script to handle a single 301 redirect to an HTTPS URL with the new site structure.

In any case, a few weeks ago, I enabled HSTS. Now, it doesn’t matter if I have “Always Use HTTPS” enabled or disabled. Either way, it issues a 307 Internal Redirect to the SSL version of the same URL.

Is there any way to revert to pre-Cloudflare behavior here?

I don’t have much experience with workers, but could a worker be issued to call the origin server for non-SSL requests?

[sandro]

I am not sure if I fully understand your question.

If you have mentioned option disabled, all HTTP requests will still hit your origin via HTTP. It is then up to you what to do with that request and whether to redirect it to HTTPS or not. Cloudflare should not perform any redirects in that case.

[DanWeb]

So that’s my question.

If the “Always Use HTTPS” option is disabled, the expected behavior is for all HTTP requests to hit my origin server.

However, that doesn’t seem to be happening.

Before onboarding to Cloudflare last month, my PHP script was successfully handling HTTP->HTTPS.

However, since onboarding to Cloudflare, HTTP->HTTPS redirections seem to be happening outside of my PHP script.

My Nginx configuration has not changed.

[sandro]

@DanWeb

If the “Always Use HTTPS” option is disabled, the expected behavior is for all HTTP requests to hit my origin server.

Precisely. If you don’t have that enabled Cloudflare won’t redirect HTTP.

@DanWeb

However, that doesn’t seem to be happening.

Would you have a link where we can reproduce that?

[DanWeb]

So my site is DaniWeb.com

So a random forum thread from 17 years ago used the URL structure:

http://www.daniweb.com/forums/thread123.html

Nowadays, the URL structure for that same thread is:

https://www.daniweb.com/hardware-and-software/networking/threads/123/netware-is-solid

Prior to Cloudflare, if there was a request for the first URL, my PHP script would issue a 301 redirect to the second URL. It would upgrade the SSL status as well as transform the URL structure at the same time, with a single 301 redirect.

Upon initially switching to Cloudflare, the behavior was such that:

• If Always Use HTTPS was enabled, it would issue a 301 redirect to https://www.daniweb.com/forums/thread123.html Then, my PHP script would issue a second 301 redirect to the proper current URL.
• If Always Use HTTPS was disabled, it would issue a 307 redirect to https://www.daniweb.com/forums/thread123.html Then, my PHP script would issue a second 301 redirect to the proper current URL

Over the past two weeks, I’ve enabled HSTS. Now the behavior is:

• Regardless of whether Always Use HTTPS is enabled or disabled, it issues a 307 redirect to https://www.daniweb.com/forums/thread123.html as so:

    Request URL: http://www.daniweb.com/thread123.html
    Request Method: GET
    
    Response Headers:
    
    Status Code: 307 Internal Redirect
    Referrer Policy: no-referrer-when-downgrade
    Location: https://www.daniweb.com/thread123.html
    Non-Authoritative-Reason: HSTS
    
    Request Headers:
    
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
    Upgrade-Insecure-Requests: 1
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36
  

And then, of course, after that, is a second redirect issued by my origin server from https://www.daniweb.com/forums/thread123.html to the proper (current) URL for the page.

My /etc/nginx/nginx.conf file, as well as all .conf files inside /etc/nginx/conf.d do NOT have: add_header Content-Security-Policy upgrade-insecure-requests; or anything that looks like it would be doing any non-HTTPS to HTTPS redirects.

Additionally, my Nginx configuration has not changed since onboarding onto to Cloudflare.

I understand that I’m using HSTS now, but when I first switched to Cloudflare, I was not, and it wasn’t working then either.

My server was properly handling SSL requests for many years, and up to the minute before onboarding to Cloudflare.

Thanks for any help!!

[sandro]

One single link would have sufficed

So http://www.daniweb.com/forums/thread123.html should be a 301 to the new link, right?

[DanWeb]

Sorry, I edited my post and put the links within code tags because Discourse was following the links, processing the redirects, and transforming the URLs.

[sandro]

I don’t get a 307. I get a 302, however that is the same what I get from your own server (assuming its IP address ends in 107).

[sandro]

Do you have anything in front of Nginx? HTTP responses do not contain a server header nor the PHP header, HTTPS responses however do. That would make me think HTTP requests do not even get to Nginx but are redirected before it.

[DanWeb]

We do have a load balancer in front of Nginx, but everything worked with no issues before Cloudflare. Literally the only difference between our PHP script handling the redirect or it being circumvented before getting to our PHP script was changing our domain DNS entries to Cloudflare.

[sandro]

Then I’d check the load balancer. The redirect in this case does not come from Cloudflare.

$ curl -I --resolve www.daniweb.com:80:BLAHBLAH http://www.daniweb.com/forums/thread123.html
HTTP/1.1 302 Found
Cache-Control: no-cache
Content-length: 0
Location: https://www.daniweb.com/forums/thread123.html

[michael]

The 307 is an internal redirect due to the HSTS header.

In which case, you may have users with your site on the HSTS list in their browser (from a previous visit), but clicking on old http URLs, but visiting you over HTTPS. It would be much easier to do a blanket HTTPS redirect in cloudflare, and redirecting (perhaps a second time) to the new canonical URL.

Also, your HSTS max age is too low to be preloaded, but you are issuing a preload command.

[sandro]

Unrelated, but because I noticed it.

You are still running PHP 7.2.19. That version has been unsupported for almost a year and 7.2 will be entirely out of support at the end of this year. I’d recommend to upgrade to 7.4 soonish.

[DanWeb]

Hi,

This is all a little bit over my head so I reached out to my sysadmin.

Basically he confirmed that our load balancer (which sits in front of the web server, and lives at .107) is doing 302 redirects. I don’t understand why this is, because I swear that pre-Cloudflare, my PHP script was handling SSL redirection. He doesn’t understand how my PHP script ever handled SSL redirection, because he swears he hasn’t recently changed any load balancer settings. The result of our debate is that he’s going to investigate why the load balancer is doing it and how to get it to stop moving forward.

However, that still doesn’t resolve the HSTS issue. When I go to http://www.daniweb.com, I am getting a 307 Internal Redirect due to HSTS. Is it not possible to have the security advantages of HSTS but also not do two redirects in a row?

I don’t know what you mean about my HSTS max-age being too low?

My Cloudflare settings say:

HTTP Strict Transport Security (HSTS)

Enforce web security policy for your website.

Status: On
Max-Age: 6 months (Recommended)
Include subdomains: On
Preload: On

Can you explain to me why six months isn’t ideal? Are you referring to my HTTP prefetch headers? Your explanation is a little over my head. I’m a web developer, but I don’t have much experience with anything that lives outside my little PHP/MySQL bubble.

[sandro]

You don’t really need to worry too much about HSTS, unless you planned to move back to HTTP. The preload issue is only because you specified half a year, whereas https://hstspreload.org requires a year.

The HSTS seems to be a Chrome peculiarity as that redirect does not show up in Firefox at all, but Firefox goes straight for HTTPS.

What I’d recommend is to have Cloudflare perform the HTTPS redirect and only have the redirect from the previous to the new URL structure on your side.

[DanWeb]

The preload issue is only because you specified half a year, whereas https://hstspreload.org requires a year.

I have no experience with HSTS other than clicking a little checkbox in Cloudflare. Cloudflare says 6 months is recommended, so I went with their default setting. I see what you mean about hstspreload.org. Since my setting is set to 6 months, and I just enabled it 2 weeks ago, then it seems highly unlikely that my domain is included in Chrome’s HSTS preload list. That being said, if 1 year is recommended for the full benefit of HSTS, then perhaps Cloudflare should make 1 year their recommended setting instead of 6 months?

My sysadmin has corrected the issue with the load balancer so that it is no longer part of the equation. If you go to http://x.x.x.107/forums/thread123.html you are now properly redirected to https://www.daniweb.com/hardware-and-software/networking/threads/123/netware-is-solid in a single 301 redirect.

However, if I go to http://www.daniweb.com/forums/thread123.html, I’m still getting the 307 Internal Redirect thing which causes the double redirect.

[sandro]

Enabling HSTS should be done with care as that glues you to HTTPS. As long as you stay with HTTPS it isn’t much of an issue. Should you ever want to move away, that is when it gets tricky.

@DanWeb

I’m still getting the 307 Internal Redirect thing which causes the double redirect.

That link doesn’t give me a 307 but a 301 which seems to come from your server. If you really get a 307 then this will be most likely a “local” redirect by your browser. You aren’t use Firefox, are you?

[DanWeb]

Enabling HSTS should be done with care as that glues you to HTTPS. As long as you stay with HTTPS it isn’t much of an issue. Should you ever want to move away, that is when it gets tricky.

We have been using HTTPS-only for the past 7+ years, so I can’t think of a reason why we would ever want to go back. The concern here is that my website is 20 years old, and has accumulated quite a handful of links from other sites over the years.

In analyzing Googlebot’s crawl the other day, I’ve noticed that it’s doing an additional 5K redirects a day due to the double redirect issue going on here. Instead of Googlebot being redirected 5K times a day, they’re having to process 10K redirects a day. From an SEO perspective, this is bad for my crawl budget. Additionally, it takes Googlebot a lot longer to process URLs when multiple redirects are involved.

Now that the issue with the load balancer is corrected, I’m hoping that Googlebot won’t encounter the HSTS issue that I’m encountering which is still causing the double redirects … or will they, since Googlebot’s crawler is just a headless version of Chrome?

[sandro]

The redirect you are referring to is not a redirect in the first place as that is something the browser runs internally. Firefox for example doesn’t show it.

Try it with a different browser or reset your Chrome and the first request shouldn’t show a 307.

[DanWeb]

I see that Firefox is not showing a double redirect.
Safari is doing a 302 Found and then a 301.
Chrome is doing a 307 Internal Redirect and then a 301.

What’s up with Safari?!

Thank you for all your help.