URLs being rewritten, slashes removed

We have an application sitting behind CloudFlare that is receiving incorrect URLs after they go through CloudFlare. For example, we have a URL at https://cloudflaredomain.com/test=/https://anotherdomain.com/test.jpg?format=webp, that our application is receiving with the path /test=/https:/anotherdomain.com/test.jpg?format=webp (notice the missing / in the protocol). We have only started seeing this issue in the past 8 hours and is causing issues with the loading of some images on our website.

Is anyone else experiencing anything similar, or has there been any changes recently that could have caused this behaviour?

Thanks!

Please turn off “URL Normalization” then it will work properly.

Quote from: https://developers.cloudflare.com/rules/normalization

// becomes /

2 Likes

Thanks! I should have mentioned, our application logs yesterday were not seeing this issue, ie. it had the full https:// as part of the path when it reached our application. We gave it a go by turning off URL normalisation, but it doesn’t seem to have had any effect, unless it can take a while to propagate.

Should go fairly quick. Please show/tell which of these you have turned on and off:

Please clear the cache for this URL https://cloudflaredomain.com/test=/https://anotherdomain.com/test.jpg?format=webp on:

  1. Cloudflares Edge (best would be complete “Purge Everything”)
  2. on your browser, as it triggers a 301 redirect, which will be caches on client side aswell.

Then try again

1 Like

Hello,

We started seeing the same problem starting yesterday. Our URLs look like: https://via.hypothes.is/https://example.com/foo.pdf?bar=1. Yesterday afternoon we starting seeing errors and noticed in our application logs that the received HTTP requests had paths that look like /https:/example.com/foo.pdf?bar=1. Note the removal of a slash after https:/.

We noticed in the process that the issue only seemed to happen if the URL contained a query string. If there was no query string, the path was not affected.

We had not made any changes to URL normalization in Cloudflare. Our initial settings were:

  • “Normalize incoming URLs”: On
  • “Normalize URLs to origin”: Off

We have tried turning the first setting off and on and confirmed that it doesn’t appear to make a difference.

You can reproduce the issue in our service by visiting https://via.hypothes.is/https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf?q=1. This will display an error dialog showing the URL received by our app (in “Error details”). If you visit the same URL with the query string omitted, our app receives the expected URL and everything works.

Is this a Worker? If so, this is a known bug.

We’re not using Cloudflare Workers. This is interesting to know though - do you have a link with more details?

Like mentioned above, you will need to:

  1. purge your Browsers cache for this URL (or test on anonym tab)
  2. purge that particular URL from Cloudflares cache

Then try again. For me btw the rewrite & normalization is not getting applied on your testURL. So I think deactivating Normalizing URLs did work. I also noticed the “BYPASS” header on your site. Have you used any PageRule to bypass this?

The main cause for this problem is I think the implementation of this “URL Normalization” as it works with a rewrite to the normalized versio, on top also a 301 which will be cached nearly everywhere… in the FAQ there was nothing written about this and also in other documentations I have not found anything about this.

By reading the FAQ I assumed it is an internal normalization, not a redirect and therefore a new request from the client. Also by starting a new request the original request (and therefore the “un-normalized” URL) will be lost and IMHO can never be applied/send to the origin, even if you would activate the second option!

Thats still true. With query string it does not work for me aswell.
Please notice that the Error details expose what is going on:

with a query string the URL will be like this:
image
https:/www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf?q=1

notice the “:/” which is not valid sheme/host and therefore will not work. I think the devs are working on this. Give them some time.

Hi @M4rt1n - Thanks for the response. The triple slash shown prominently on the error page above is an artifact of how our Python app processes URLs that start with http:/hostname/... internally (note single slash). The single slash in the “Error details” panel shows what our app is actually receiving.

Given the fact this behavior changed yesterday it certainly seems like this is a bug caused by a recent change on Cloudflare’s end.

Thanks for the clarification on the tripple slash. So it does not produce a trippleslash “:///” but a single slash “:/”.

Hi there,

An update here, we believe this was caused by a bug in a release yesterday a fix is being worked and should be out today.

Apologies for the inconvenience this caused

Thanks for the quick response, much appreciated.

A question someone else on my team had is whether there is any other kind of rewriting is being applied to URLs when URL normalization is off, aside from this unintentional change. We’ve not experienced any problems up until now, but is there anything we should be aware of?

Hi checking in on the status of this fix. Has this issue been resolved?

Thanks.

The test URL I linked above is now working, so it looks like the issue has been resolved. We put a workaround in place for our site that we plan to undo on Monday morning, so I’ll be able to confirm more thoroughly then.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.