How to debug requests through 1.1.1.1 appearing as from wrong continent?

Hello,

we are using AWS Route53’s Geolocation feature to direct users to our closest server.

When we run dig @1.1.1.1 mydomain.tld from a server in Singapore, we do not get the response configured in Route53 for Asia; instead we get the response configured for North America.

I understand that 1.1.1.1 does not support ECS. However, 1.1.1.1 is 0.7 ms away, thus also in Singapore, so I would still expect that Route53 sees the DNS request coming from Cloudflare as originating from Singapore.

This thread confirms that this is how it should work, and when it does not work, it is probably a wrong entry in some (AWS’s?) geo location database that misplaces Cloudflare’s IPs.

How can I debug whether this is the case? How can I get it fixed? If the problem is in Route53, how can I determine from which IP 1.1.1.1 will make the request to Route53, so that I can report it to AWS support?

Currently our users are sent around half the globe when they use 1.1.1.1, which leads to a bad experience.

Thanks

The problem here is the lack of ECS support.

Cloudflare routes based on one single anycasted IP address. Amazon apparently based on ECS and returns different IP addresses based on the client’s location.

As long as Cloudflare does not support ECS or Amazon uses a single anycasted IP address as well, you wont be able to fix this using DNS only.

I believe this is incorrect. As I already stated in the the issue description (and linked another post that confirms that):

I think this is based on a misunderstanding. Cloudflare uses anycast to answer DNS requests, but Cloudflare apparently uses “normal” IPs to make DNS requests.

Amazon’s docs state:

If this [ECS] data isn’t passed with the request, Route 53 uses the source IP address of the DNS resolver to approximate the location of the client and responds to geolocation queries with the DNS record for the resolver’s location.

This means Route53 will use the location of the Cloudflare machine that talks to it.

Based on what? What in your posting contradicts any of what I said?

That is what I said.

That is how it would seem, but if that address does not scream “Singapore” Amazon will still route to whatever it associates with that address and that seems to be the US.

You can easily check that by using any of the available DNS leak tests and check where the server’s IP address appears to be registered.

My understanding is that you were saying that this can only be fixed if Cloudflare starts supporting ECS.

But I believe it would be sufficient if the IP from which Cloudflare talks to Route53 had correct geolocation DB entries (then, Cloudflare’s Singapore resolvers would show up as in Singapore to Route 53).

Your last post (“that address does not scream ‘Singapore’”) seems to agree with that, but your first post seemed to suggest something different – maybe I misunderstood you?

I got a bit further, also managing to confirm how the Cloudflare IP appears to Route53.

AWS offers functionality to report the IP that is talking to it:

  1. Check the client’s DNS resolver to verify that it is geographically close their location. Route 53 uses resolver-identity.cloudfront.net to reflex the IP address where the DNS query came from.

When I dig @1.1.1.1 resolver-identity.cloudfront.net, repeatedly, I get these IPs returned:

172.69.132.8
162.158.161.159

These appear to be in the Cloudflare IP ranges as expected.

Now the issue seems to be that indeed, different geolocation DB providers disagree on where those IPs are based:

  • https://www.maxmind.com/en/geoip-demo (which the AWS doc recommends) classifies them as in the US
  • IPAPI, (e.g. curl "https://ipapi.co/172.69.132.8/country/" and curl "https://ipapi.co/172.69.132.8/country/") classfies them as in SG

Now the question is: Should these IPs have a single “correct” geolocation?

If yes, the Maxmind DB seems wrong, and probably whatever AWSs DB is also has it wrong then. If that is the case, whose task is it to correct these entries?

Correct, that or Amazon uses Anycast as well.

“Correct” as in? The upstream addresses do not necessarily need to correlate with the country where you sent the request from. a) Cloudflare wont have addresses “from” each single country where they have a datacentre. b) Even if they did, there is no guarantee you’d be routed to that datacentre.

There is nothing “correct” about country allocations when it comes to IP addresses, I am afraid. There is just a “less wrong” :slight_smile:

ARIN lists the address in question as US based. It is unclear where IPAPI got Singapore from but that does not seem to be necessarily authoritative nor necessarily correct.

Yes, that makes sense from a technical perspective.

I mean “correct” as in “what Cloudflare likely intends to happen”.

It appears to me that especially given that Cloudflare does not support ECS, it would probably want the resolver’s IP to count as “reasonably close” to the user – at least in the same country, so that issues as I am seeing do not happen and users of 1.1.1.1 do not have a bad time.

APNIC, for example, suggests on https://www.apnic.net/get-ip/faqs/geolocation/#updates that:

… Members who hold “allocated portable” IP ranges are free to create more specific inetnum and inet6num records that contain different “country” values to indicate the economy in which those IP addresses are used. Additionally, the “geoloc” attribute can be added to associate a latitude/longitude coordinate pair with the record.

and further

If you find that a geolocation provider has incorrect location
details of your IP address range, you can contact them and request they
update the location of the range.

Would it not be in Cloudflare’s interest to make their resolver IP appear from the correct country then?

Please refer to my previous postings, as I already addressed these bits.

Also, https://ipapi.co/1.1.1.1/ claims 1.1.1.1 is in Down Under :slight_smile:

Sorry, I don’t quite follow – which part are you referring to? My point was that Cloudflare could choose to make their Singapore resolver server’s address like 162.158.161.159 show up as in Singapore in common geolocation DBs. Your posts make clear that this would not happen automatically, but I cannot infer from them why Cloudflare would not be interested in updating them explicitly.

1.1.1.1 is used as a multicast IP though, it makes sense that it has no defined location. 162.158.161.159 does not appear to be one. Also, 1.1.1.1 does not talk to upstream DNS resolvers, while resolver IPs like 162.158.161.159 do.

To this part

The point I was trying to make is that IPAPI’s data is not necessarily accurate.

Thanks. I agree that geo IP databases are not necessarily accurate without manual invervention, but as mentioned in the APNIC snippet further up, one can contact them and request to update the DB.

I guess a statement from Cloudflare would be helpful on whether this is something they would like to do.

The accuracy of IP databases is one part of it, the other is simply what I mentioned earlier. These addresses are all part of a larger block and you cant “assign” countries to individual addresses.

On top of that, as I already mentioned, you wont even have a guarantee that you will be routed via Singapore. Even if Cloudflare assigned to each address the respective country (what about countries where they dont have a datacentre?) you’d still be “incorrectly” routed if you go via a different country.

So, again what I said in the beginning, either Cloudflare needs to support ECS or Amazon needs to switch to Anycast.

1 Like