Azure web app to web app calls fail with 403 when both using Cloudflare SSL

ssl
tls

#1

Since July 19th I’ve encountered an issue with Azure web apps calling each other server side, something that previously worked flawlessly now fails with 403 errors. I believe it’s likely to be related to their recent TLS 1.2 initiative mentioned in this blog post - https://blogs.msdn.microsoft.com/appserviceteam/2018/05/02/breaking-change-for-sni-ssl-hostnames-on-azure-app-service/

The problem occurs when I’m doing server-side http requests (browsers not involved). I’m only encountering the issue when both endpoints are using custom domains employing Cloudflare so I’m reaching out in the hope someone can help me understand what the issue might be and maybe help me arrive at a solution.

An example of the failing call:

https://www.topomap.co.nz/proxy.ashx?https://www.permitwatch.co.nz/api/nzpam/GetKmlFile?type=101&primaryKey=60481.01&updatedUtc=2018-08-10%2015%3A01&test

topomap.co.nz is making a request server-side to the url provided in the query parameter, permitwatch.co.nz in this case.

topomap.co.nz is using Cloudflare’s Full SSL, with the certificate loaded server side in Azure being the same wildcard domain certificate used by https://tiles-1.topomap.co.nz/ (my map tiles are hosted elsewhere on a non-Azure service). In Azure the binding is SNI SSL.

permitwatch.co.nz is using Cloudflare’s Flexible SSL option. No other ceriticates involved.

If I replace either topomap.co.nz or permitwatch.co.nz with an some-app-name.azurewebsites.net equivalent (the default url used by Azure before adding a custom domain) everything works seamlessly again.

The 403 seems to be coming directly from permitwatch.co.nz as there’s no Cloudflare branding on the message displayed.

I can use https://www.topomap.co.nz/proxy.ashx to call out to any other non permitwatch.co.nz without issue e.g.

https://www.topomap.co.nz/proxy.ashx?https://www.dropbox.com/s/hx2supar5eonx5x/example.kml?raw=1

And just to provide a bit of background - I’m proxy’ing files to avoid CORS issues for people wanting to overlay GPS tracks etc. on to the map. People who mostly lack the technical expertise to setup CORS their side of things e.g.

https://www.topomap.co.nz/NZTopoMap?v=2&kml=https%3A%2F%2Fwww.dropbox.com%2Fs%2Fhx2supar5eonx5x%2Fexample.kml%3Fraw%3D1

Thanks in advance for any insights that anyone can provide.

Update:

I’ve put in place a horrible hard-coded hack in place (redirecting to alternative urls) so the following will now work:
https://www.topomap.co.nz/proxy.ashx?https://www.permitwatch.co.nz/api/nzpam/GetKmlFile?type=101&primaryKey=60481.01&updatedUtc=2018-08-10%2015%3A01

but to still test without the hack in place &test can be added to the end of the url:
https://www.topomap.co.nz/proxy.ashx?https://www.permitwatch.co.nz/api/nzpam/GetKmlFile?type=101&primaryKey=60481.01&updatedUtc=2018-08-10%2015%3A01&test


#2

Maybe a few Cloudflare users out there could help me out?

Microsoft Azure support are telling me the 403 is from Cloudflare even though it doesn’t carry any Cloudflare branding (but don’t seem to have any solid evidence to backup the claim), so perhaps some of you might be able to help me out prove this one way or another…

I just want to see if you can use the proxy on some static asset from your site using my service, such as an image or text file. Simply add the static asset url after the proxy.ashx? e.g.

https://www.topomap.co.nz/proxy.ashx?https://www.paydirt.co.nz/Discourse/nugget-144x144.png

I believe this fails with a 403 also as paydirt.co.nz is on Azure and is behind Cloudflare Flexible SSL.

I’d be curious to see a mixture of static asset tests to try and zone in on the particular variables that may be causing the 403, such as:

  • Site hosted on Azure / not on Azure.
  • Different Cloudflare SSL options being used.

Any help would be greatly appreciated. Thank you in advance.


#3

Could it be the IP range you are sending the request from is blocked on Cloudflare’s side?

I am not sure TLS is involved at all, I just checked https://www.topomap.co.nz/proxy.ashx?http://www.sitemeer.com and got the same message. Do you have any chance to send an HTTP request manually from the server in question?


#4

@sandro I don’t believe the IP range is being blocked on Cloudflare’s side as if I remove my custom domain I can get past the 403 e.g.

https://testing403.azurewebsites.net/proxy.ashx?http://www.sitemeer.com/logo.png (I get a 503 for your site

https://testing403.azurewebsites.net/proxy.ashx?https://www.paydirt.co.nz/Discourse/nugget-144x144.png (works)

I’m also using Cloudflare’s Full SSL option for topomap.co.nz and Flexible SSL for permitwatch.co.nz, so I’d assume the IP range couldn’t be blocked by Cloudflare?


#5

Considering that paydirt is behind Cloudflare as well, it actually seems unlikely it is blocked on Cloudflare’s side.

Well, in this case you’d really need to check what your proxy code is actually doing on the server side. Do you have access to that? And can you issue an HTTP request manually from there?

But we can rule out a TLS issue, cant we.


#6

I’ve access to the code so I have tried various different methods there to try and get past the issue, with no joy. I’ve captured trace logs for the failed 403 requests, but they just show a 403 without sub-codes or any further explanation.

Regarding getting onto the IIS server itself, this is impossible from my side as it’s an Azure Web App (managed service, rather than being a virtual server you can directly remote in to).


#7

Alright, do you think you could deploy a simple script which opens a TCP socket and connects to e.g. sitemeer.com on port 80 and sends a plain HTTP request? If that works there must be a problem with the code in the ASHX file. If not, there is some network issue involved.


#8

Using C# that is the crux of what is going on in proxy.ashx. The code seems to be working fine for any non-Azure based site, so I believe the issue must be with Azure’s network.

I’m wondering if the Azure network is getting upset at possibly finding 2 different SSL certificates for www.topomap.co.nz? …

On Azure www.topomap.co.nz has the same AlphaSSL CA certificate loaded against the service as is used here - https://tiles-1.topomap.co.nz/ (I host the tiles in NZ for reduced latency). This is so I can use Cloudflare’s Full SSL option.

But to the ouside-of-Azure-network world www.topomap.co.nz is providing a COMODO certificate issued by Cloudflare.


#9

The HTTP request failed too, hence I believe we can rule out TLS.

It would be important to know where your proxy script connects to and what it actually receives from there.


#10

Good point regarding the http / TLS. So you doubt it will be Azure services rejecting calls (via a 403) from topomap.co.nz because of the 2 different SSL certificates present for topomap.co.nz?

Regarding the proxy code, I’m using this (configured to allow proxy’ing to anything) …

There’s a lot of extra stuff in there I don’t need, but I’ve stepped through the code locally and can see it’s just re-issuing a basic GET request for these calls by-passing all the more advanced logic.


#11

I would check if the error you receive actually comes from the origin (or something inbetween) or is generated by the script. If it is the former, something blocks your request and, as it does not appear to be Cloudflare branded, my guess would be there must be some sort of transparent proxy inbetween your code and Cloudflare.


#12

@sandro Thanks for your input proding my thought process, it does seem that any branding present might be getting masked, if it is indeed present.

I tried the proxy to a 3rd party pretty 404 page. The code should have been returning the full content, but I wasn’t seeing it when run locally. I made some changes to the proxy code which provided the full content without issue when tested locally. However, when I deployed to Azure it again isn’t forwarding any content, just the response code. So a bit more investigation for me to do my side first it seems.


#13

Maybe a stupid question, but just to be sure and because it already happened to me too :slight_smile: … have you double checked that it executed the newly deployed code? Maybe you can insert a temporary additional HTTP header returning some version code, for you to verify you are on the version you expect.


#14

I must admit, it occurred to me also. I checked the script file on the server and it was definitely the updated one. Still makes me wonder if an old version is getting cached somehow, and if so, that will call into question some of my previous experiments.

The proxy script does sit on it’s own outside of the usual compiled .NET dll’s for the app, so maybe it’s getting treated differently. Shouldn’t be. It’s also been nearly 24 hours since I deployed so app pools should have recycled several times by now etc. Next chance I get I’ll bring the code into the fold of the rest of the solution that gets compiled up every time deployed and re-test anyway.


#15

My experience with .NET is far too mediocre to give you useful advice here :slight_smile: but maybe try the thing with the HTTP header. Have some version information shown when you request the root path should work too.

But as for the issue, could you deploy a simplified version of it which really only tunnels through data (my understanding is the code right now does a bit more). To go a final step further - and complete total and utter paranoia :slight_smile: - maybe you can log all crucial information to a log file too, just in case there is not only a proxy between your script and the origin server but also one between you (the client) and the script. No idea if any of that might even be the case, but just to rule it out. Comes to show how well I know Azure’s environment :laughing:


#16

From experiments it seems that as soon as an app hosted in Azure returns some http error type status code, then all content is stripped off. So thins was causing my confusion about the source of the error.

It does seem in the end that the proxy.ashx handler from ArcGIS was incompatible with Azure. I’ve since swapped it out for a much simpler one I wrote from scratch and it seems to work fine so far. I should have listened to you much earlier regarding trying a simple test first! Thank you for your perseverance :wink:


#17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.