Cloudflare orange cloud and IP

Website, Application, Performance DNS & Network
1

I am going to setup an Azure Traffic Manager that is going to point to three regions. In cloudflare I have a CNAME to the traffic manager. The traffic manager points to three domains also controlled by Cloudflare, but Cloudflare just gives an error 1000. Alle three region CNAME is orange clouded. The traffic managers CNAME is gray clouded. If I gray cloud the three region CNAME it works, but I really don’t get any caching benefits then from Cloudflare. The strange thing is, if I orange cloud the CNAME to the traffic manager and gray cloud the region CNAME, it works, but then my traffic manager don’t my end users IP, but what ip is provided then? Is it the IP of the closest exit node to the end user? Or?

Another question is the error 1000. It don’t get why it doesn’t work either way.

2

Hi @mslot,

AFAIK, you can’t point a :orange: to another :orange:.

Error 1000: DNS points to prohibited IP

Common causes

Cloudflare halted the request for one of the following reasons:

  • An A record within your Cloudflare DNS app points to a Cloudflare IP address.
  • Your Cloudflare DNS A or CNAME record references another reverse proxy (such as an nginx web server that uses the proxy_pass function) that then proxies the request to Cloudflare a second time.
  • The request X-Forwarded-For header is longer than 100 characters.
  • The request includes two X-Forwarded-For headers.

Resolution

  • If an A record within your Cloudflare DNS app points to a Cloudflare IP address, update the IP address to your origin web server IP address.
  • There is a reverse-proxy at your origin that sends the request back through the Cloudflare proxy. Instead of using a reverse-proxy, contact your hosting provider or site administrator to configure an HTTP redirect at your origin.

Troubleshooting Cloudflare 1XXX errors

About end users’ IPs, take a look at this Support page:

https://support.cloudflare.com/hc/en-us/sections/200805497-Restoring-Visitor-IPs

3

And that is actually what baffles me, because I am not pointing a :orange: to a :orange:, but a :cloud: to a :orange::

image
image2141×304 39.7 KB

As I understand it, this is how it works:

  1. subdomain.domain.com :cloud: traffic manager - traffic manager sees my end users real IP, and then,
  2. the traffic manager in Azure determines which is the best region (it looks up in some predefined rules). I my case it just validates EU to,
  3. tmtesteu.domain.com, that CNAMES a :orange: to a azure website

Isn’t that how :gray: is understood? Or am I missing something?

If we reverse the :gray: to a :orange:, well then the traffic manager sees the resolved exit node of Cloudflare, which has been determined to be closest to the user, or am I wrong about this?

A side note: it actaully works pointing :cloud: to :orange: for about five minutes. And then it shows me ERROR 1000. I dont know if this has something to do with Azure Traffic Manager, but in my mind this should work, because i am not pointing anything through Cloudflare with my :cloud:.

Maybe I am seeing a double X-Forwarded-For from the traffic manager? I doubt it, because the Traffic Manager, as i understand it, doesnt act like a reverse proxy, but just resolves DNS.

Traffic Manager uses DNS to direct clients to specific service endpoints based on the rules of the traffic -routing method. Clients connect to the selected endpoint directly. Traffic Manager is not a proxy or a gateway. Traffic Manager does not see the traffic passing between the client and the service.

From the Traffic Manager docs: How Azure Traffic Manager works | Microsoft Learn

4

:grey: means that Cloudflare is acting as DNS-only for this record, not proxying it.

Not necessarily the closest, as this may depend on numerous factors (including subscription plan).

If so, yes. This is one of the situations described among the common causes.

You said about reversing :orange: and :grey:. Was that done for all the records?

I’m not familiar with Azure and Traffic Manager, so I won’t be able to help you with specific issues.

5

Azure Traffic Manager uses DNS to direct a call to a specific endpoint, so I dont think it is the X-Forwarded-For that is the problem.

My setup right now is

image
image2070×315 40.6 KB

so tmtest.sigsys.dk is pointing to the traffic manager, which looks at my IP and then sends it to the closest region (I have some rules set up). In my case, it is tmtesteu.sigsys.dk, which is :orange: in Cloudflare.

So I am pointing a :cloud: to a :orange:. Shouldn’t this work?

6

AFAIK, yes, It should work.

Unfortunately, this is going beyond my knowledge, so I’m afraid we’ll need a more experienced user to help us.

I saw that @sdayman recently published, maybe he can bring some light to this issue.

7

It could be extremely helpful for a more experienced user to look at this. @sdayman, do you know more than we do?

8

The way this is set up, the user always gets the same result from the Azure Traffic Manager,

% dig tmtest.sigsys.dk
tmtest.sigsys.dk.	92	IN	CNAME	azurewebapptest.trafficmanager.net.
azurewebapptest.trafficmanager.net. 35 IN CNAME	tmtesteu.sigsys.dk.
tmtesteu.sigsys.dk.	92	IN	A	104.18.63.40
tmtesteu.sigsys.dk.	92	IN	A	104.18.62.40

The user then connects to one of those two IP addresses, and asks the Cloudflare cache for tmtest.sigsys.dk. The origin for this is azurewebapptest.trafficmanager.net., which resolves to the same two Cloudflare IP addresses, and you get the error 1000 as this is :orange: on :orange:.

I have not thought about this enough to be definitive, but if you :orange: the ‘user facing’ name, and :grey: the other three do you get what you expect? (With DNS timeouts you might have to wait a few mins to see a change)

If you are trying to fill the Cloudflare cache as efficiently as possible, then it does not matter if Azure sees the users IP address or not. It is the Cloudflare POP you want to get the best result, lowest latency etc. I’m not sure Azure will be able to tell where the POP is though, so you might not be able to use ATM here to get your desired outcome.

It seems from the OPs comments that they want the users to get their content from a Cloudflare cache, so what matters is that the user gets the best Cloudflare POP (outside the OPs control) and that the Cloudflare POP gets the best Azure location. Azure does not need to know the users IP address, and that might actually make the outcome worse.

A Cloudflare load balancer with Dynamic steering (fastest pool) will give you the result you want, but you will have to run the numbers.

2 Likes
9

Thanks for filling in. I can’t see how it is :orange: to :orange: because tmtest is :cloud: and tmtesteu is :orange:.

Because tmtest.sigsys.dk (:cloud:) - - > azurewebapptest.trafficmanager.net - - → tmtesteu sigsys.dk (:orange:)

So I can’t see the :orange: to :orange:. This morning it actually worked for two requests, and then it stopped again.

I have doubts reverting this to a :orange: tmtest.sigsys.dk and :cloud: the others because I have understood that the IP sent to traffic manager fra Cloudflare might been totally off (which I actually don’t understand either because, well, wouldn’t CF try to resolve the best possible POP closest to the user which I then use to route to an Azure datacenter?).

10

The Origin IP can never be a Cloudflare IP address. This is what is called the orange-to-orange problem.

When a user wants to visit tmtest what do you expect to happen? What IP address do you expect Cloudflare to use when it wants to connect to the origin for tmtest? How have you configured the DNS to respond accordingly? Do you want users to connect to the Cloudflare cache for tmtest, or to the Azure web server?

Break the HTTP request into two parts: the DNS lookup, and the CDN fetch.

With tmtest :grey: the DNS lookup goes like this, client ask for tmtest and is told by Cloudflare:
tmtest.sigsys.dk. IN CNAME azurewebapptest.trafficmanager.net.

The users device then looks up azurewebapptest.trafficmanager.net, which is configured to return one of tmtesteu, tmtestusa, tmtestau, depending on how the Azure Traffic Manager is configured. So the ATM picks one, and tells the user: the result you want is available at tmtesteu (or whatever). This happens without an involvement of Cloudflares DNS.

The client then looks up tmtesteu against Cloudflares DNS, and as that is :orange: Cloudflares DNS returns two of Cloudflares anycast IP addresses.

Regardless of which one of the three is returned, they are all :orange: so will return two A records from Cloudflares anycast range.
tmtesteu.sigsys.dk. 92 IN A 104.18.63.40
tmtesteu.sigsys.dk. 92 IN A 104.18.62.40

So the Azure Traffic manager has done nothing really. The IP addresses the client is being told to connect to will always be the same, a pair of Cloudflare anycast IP addresses with no mention of the Azure region specific IP address. It does not matter where the user is, the result is always the same.

So the client now has an address to start the CDN Fetch. The client connects to one of the Cloudflare IP addresses, and makes a HTTP request for Host: tmtest.sigsys.dk. Cloudflares cache receives the request, and Cloudflares servers are now acting as a client. The Cloudflare cache says “the origin is azurewebapptest.trafficmanager.net, what’s the IP for the origin”. Repeat the process above, and Cloudflare determines that the origin IP address is the same anycast IP address pair. The origin address cannot be a Cloudflare address, :boom: error 1000.

If you reverse the clouds, the first lookup will be:
tmtest.sigsys.dk. 92 IN A 104.18.63.40
tmtest.sigsys.dk. 92 IN A 104.18.62.40

The client will connect to a Cloudflare cache. The cache will resolve azurewebapptest.trafficmanager.net, ATM will use the Cloudflare source IP in its decision making process, return one of the three region specific CNAMES, which will in turn return the region specific Azure IP addresses. The Cloudflare cache will then connect to the Azure IP, and fetch the asset, populate the cache, and return an asset to the client. Everybody is happy? The only issue here is that I do not think the Cloudflare source IP will contain any region specific information for ATM to make a valid decision. A region specific address on the Cloudflare cache would be an attack vector to a particular cache, and I do not think that is likely. My guess is that you will always get the same Azure region in the DNS response from ATM to Cloudflare.

11

Thanks for the breakdown!! I kind of understand. As you may have guessed, I am not a “network guy”, I am just a developer interested in Cloudflare and Azure. Your answer really helps me. Thanks for providing me with this, taking your time. It really means a lot.

The reason why I did this by :cloud: the tmtest, was that i thought it would make Azure Traffic Manager better at deciding where the user resides, and then, afterwards, I wanted Cloudflare to cache accordenly.

I have just tried to reverse the clouds:

image
image1887×337 37.7 KB

Now, as I see it, I:

  1. dont get a error 1000
  2. i am sitting in europe, and get a response from the site that resides in europe
  3. i get the ability to cache through Cloudflare, (or am i missing something else fundemental?)

if I run it through webpagetest.org, from USA, I get the site that resides in USA: WebPageTest Test - Running web page performance and optimization tests...

This makes me wonder how it Azure Traffic Manager resolves this correctly, because it sees the Cloudflare anycast IPs. Maybe Traffic Manager looks on some other headers to determine the IP? CF-Connecting-IP, or X-Forwarded-For? Interesting :slight_smile:

12

ATM is a pure DNS load balancer, so no special HTTP headers can be seen.

Which traffic routing method have you chosen for ATM?

Do you have logs from Azure about what IP addresses did DNS lookups in each region?

13

I am using the geographic routing method, which mean (please correct me if i am wrong) that CF is resolving azurewebapptest.trafficmanager.net, and traffic manager is using what information is present in the DNS lookup:

Traffic Manager profiles can be configured to use the Geographic routing method so that users are directed to specific endpoints (Azure, External or Nested) based on which geographic location their DNS query originates from

I am no DNS wiz, but that is how i understand it :slight_smile:

I bet, if i switch to performance, it will always return one of the resources no matter where you are located.

14

I have tried to switch to performance on the Traffic Manager and the results is funny. Traffic Manager sees me as being close to USA, and serves traffic from there, and Cloudflare sends me rays from CPH (which makes sense because I am sitting in Denmark (CPH = Copenhagen)).

Here I think Traffic Manager uses the IPs provided by Cloudflare:

Traffic Manager looks up the source IP address of the incoming DNS request in the Internet Latency Table. Traffic Manager then chooses an available endpoint in the Azure datacenter that has the lowest latency for that IP address range, and returns that endpoint in the DNS response.

From the docs about performance mode on Traffic Manager: Azure Traffic Manager - traffic routing methods | Microsoft Learn

I will try to switch back, to test some more :slight_smile:

15

Cloudflare has PoPs in over 200 cities, all of which use the same IP ranges. If Azure is using a lookup table I would expect the same result every time. I read the ATM docs earlier, and it was not clear how the Geographic mode worked. They might know based on the Azure location that received the request ‘roughly’ where it came from.

16

With geographic mode they do look me up correctly, and all other locations i tried, ever time. With performance mode, they don’t, but they are consistently wrong everytime, and they always direct me to the same wrong place :slight_smile:

I don’t think I can test this anymore. I am doing the same with Azure Front door.

Thanks for the help, Michael! Much appreciated.

17

@mslot in the context of this topic, having gone this down road before.
Routing method:
DNS: Azure Traffic Manager
HTTP: Azure Front Door

Very similar to Cloudflare Argo, it may be easier to use Argo tbh and I’ll explain why.
Your user gets to your origin server through the CF network (Edge), which has way more PoPs the traffic needs to get into the Azure network (Cloud) which is basically a bunch of dark fiber.

Hence their Front Door and ExpressRoute offerings.

Since users are closer to the edge, it’s best to meet them there with caching and keep the workloads in the cloud.

Without knowing your exact goal, I can only suggest looking into Cloudflare Workers to aggregate requests and cache using KV storage at the edge because it is a much better user experience.

To end, the reason why your solution didn’t work. Normally an Azure App Service where your Web Apps run, don’t have have static IPs. They use the hostname from the HTTP request to find your Web App, on a higher plan you can assign a Traffic Manager.

If you’re not on a higher plan, your request will have the hostname of the traffic manager and it will fail to find your Web App.
You can only assign one TM to one ASP which only has one location. If you’re wondering what’s the point? It’s to support scaling scenarios when ASPs scales out.

If you share more, I can provide more tailored advice :wink:

18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.