Performance issue in Italy


#1

I read about 1.1.1.1 (and 1.0.0.1), great news! Lots of POPs and great speed!

Two questions/problems:

  • are there IPv6 versions of this?
  • I see performance problems though, I am in Italy, getting replies from MXP, but Google (and others) go to FRA while using 1.1.1.1, while they go to MXP using 8.8.8.8 (10 ms of latency added).

EDIT
Found the solution to the IPv6 IPs:

$ dig AAAA 1dot1dot1dot1.cloudflare-dns.com +short
2606:4700:4700::1001
2606:4700:4700::1111

About the 1.1.1.1 category
1.1.1.1 not resolving to local Google websites
Ping to google.com and google.com.au now over 200ms
Not routed to the closest data centre
#2

Cloudflare doesn’t send Client-Subnet to resolvers, so you may get a different answer from 1.1.1.1 than you would from a server which supports that EDNS feature.

EDNS Client Subnet
1.1.1.1 is a privacy centric resolver so it does not send any client IP information and does not send the EDNS Client Subnet Header to authoritative servers


#3

I saw that, but for example Google Drive has serious performance issues, I won’t be able to test for a week the actual download/upload speeds (lower 200/20 connection now, will get back to 1000/200), but I assume I won’t be able to max the connection out to a server in FRA (or even farther probably).

This will mean that I can’t possibly use/recommend 1.1.1.1 for many occasions.

Cloudflare 1.1.1.1

PING drive.google.com (173.194.76.100): 56 data bytes
64 bytes from 173.194.76.100: icmp_seq=0 ttl=43 time=29.723 ms
64 bytes from 173.194.76.100: icmp_seq=1 ttl=43 time=193.181 ms
64 bytes from 173.194.76.100: icmp_seq=2 ttl=43 time=153.552 ms
64 bytes from 173.194.76.100: icmp_seq=3 ttl=43 time=101.994 ms
64 bytes from 173.194.76.100: icmp_seq=4 ttl=43 time=173.271 ms
64 bytes from 173.194.76.100: icmp_seq=5 ttl=43 time=248.901 ms
64 bytes from 173.194.76.100: icmp_seq=6 ttl=43 time=273.614 ms
64 bytes from 173.194.76.100: icmp_seq=7 ttl=43 time=29.440 ms
64 bytes from 173.194.76.100: icmp_seq=8 ttl=43 time=29.774 ms
64 bytes from 173.194.76.100: icmp_seq=9 ttl=43 time=44.901 ms
64 bytes from 173.194.76.100: icmp_seq=10 ttl=43 time=88.806 ms
64 bytes from 173.194.76.100: icmp_seq=11 ttl=43 time=497.326 ms
64 bytes from 173.194.76.100: icmp_seq=12 ttl=43 time=177.855 ms
64 bytes from 173.194.76.100: icmp_seq=13 ttl=43 time=158.860 ms
64 bytes from 173.194.76.100: icmp_seq=14 ttl=43 time=291.957 ms
--- drive.google.com ping statistics ---
15 packets transmitted, 15 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 29.440/166.210/497.326/122.673 ms

Google 8.8.8.8

PING drive.google.com (216.58.205.174): 56 data bytes
64 bytes from 216.58.205.174: icmp_seq=0 ttl=54 time=12.876 ms
64 bytes from 216.58.205.174: icmp_seq=1 ttl=54 time=11.865 ms
64 bytes from 216.58.205.174: icmp_seq=2 ttl=54 time=12.828 ms
64 bytes from 216.58.205.174: icmp_seq=3 ttl=54 time=11.714 ms
64 bytes from 216.58.205.174: icmp_seq=4 ttl=54 time=11.721 ms
64 bytes from 216.58.205.174: icmp_seq=5 ttl=54 time=11.795 ms
64 bytes from 216.58.205.174: icmp_seq=6 ttl=54 time=11.846 ms
64 bytes from 216.58.205.174: icmp_seq=7 ttl=54 time=13.506 ms
64 bytes from 216.58.205.174: icmp_seq=8 ttl=54 time=11.787 ms
64 bytes from 216.58.205.174: icmp_seq=9 ttl=54 time=12.531 ms
64 bytes from 216.58.205.174: icmp_seq=10 ttl=54 time=11.844 ms
64 bytes from 216.58.205.174: icmp_seq=11 ttl=54 time=12.667 ms
64 bytes from 216.58.205.174: icmp_seq=12 ttl=54 time=13.207 ms
64 bytes from 216.58.205.174: icmp_seq=13 ttl=54 time=11.620 ms
64 bytes from 216.58.205.174: icmp_seq=14 ttl=54 time=11.823 ms
--- drive.google.com ping statistics ---
15 packets transmitted, 15 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 11.620/12.242/13.506/0.606 ms

#4

Adding to this conversation @cscharff, I am having serious performance issues with YouTube (having problems maintaining a 1080p video on a 200Mbps connection due to the problems discussed above). Had to revert back to 8.8.8.8 to fix.

In addition in my university (Politecnico di Torino, www.polito.it) we have a captive portal running on 1.1.1.1, wrote to them about it, received a reply the first useful day after Easter holiday (longer than you folks in the US, impressed by them actually). I am gonna write here the reply, which points to a couple of issues that will arise in other situations as well. The sort of comments were added by me to an semi-auto translated text by Google. If you need to contact them you can do so at [email protected].

Good morning,

we are aware that the IP address 1.1.1.1 is a valid IP address
and, a few days ago, assigned by APNIC to Cloudflare.

However, we do not intend to change this address within the captive
Portal of our WiFi for various reasons including:

- 1.1.1.1 is the address of a DNS server and, in the university network,
only official Polytechnic DNS servers (those assigned to the device by DHCP)
can and must be used to fully comply with the GARR AUPs:
https://www.garr.it/it/regole-di-using-della-rete-aup

### This is understandable, but still it's not your IP to use and announce.

- IP address 1.1.1.1 is easy to store and type in to allow users
to access the controller page to log out of the service.

### In addition there is a FQDN in the DNS that points to it
### and it is much more memorable.

- the IP address 1.1.1.1 is preset and strongly recommended
by the controller manufacturer (Cisco) as the IP address for the same.
Changing it may cause some problems in its operation.

### This is the big issue I see, Cisco recommends to use it for internal captive portals.

- he policy of using address 1.1.1.1 within the controller
is common to almost all controllers for WiFi networks.

### basically same thing as before.

We remain available for any further clarification.

Best regards

WiFi infrastructure support

#5

We are working with a number of providers that use EDNS to provide them information about our network to try and help improve performance while still refraining from sending client specific ECS information… it’s a process.

As for the university reply, I’ m not sure how well :facepalm: translates into Italian…


#6

Oh I believe you, don’t get me wrong, but in the meantime some things are bad probably also due the nature of the network topology in Italy (and the EU). The central node is in Frankfurt (multi Tbps node), the national main one, especially in the northern part of Italy, is on Milan, but the links between the two have reduced capacity. In addition in the last 2/3 years there has been a massive increase in very high speed internet access. I moved from <1Mbps to 1Gbps, while the majority of services have a few tens of Gbps (you included).

I didn’t even reply to the university. DNS queries to alternative DNS servers I believe they actually work, but miss the basic auth records so most things fail. Would love to try DoH, but I’m not able to set it up!


#7

I can finally get back to you @cscharff! So… From a Gbps connection I lose about 250 Mbps in download from Milan to Frankfurt.

Now I have changed my ISP (faster upload, yeah!), but Cloudflare (both 1.1.1.1 and standard CF) instead of being announced in Milan (MXP) gets announced in Rome (FCO). I lose 6 ms of latency, but I know for a fact that my ISP (ASN 8612) peers in MXP (Google goes there, for example) as does in TOP-IX and at NaMeX.

 google.com (well, Google)

                                                 Packets               Pings
 Host                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. openwrt.lan                                 0.0%   110    0.7   0.7   0.5   1.2   0.1
 2. static-213-205-53-217.clienti.tiscali.it    0.0%   110    1.7   2.1   1.6   4.3   0.4
 3. 94.32.155.133                               0.0%   110    2.0   2.1   1.6   2.6   0.3
 4. 94.32.154.5                                 0.0%   110    5.1   3.3   1.5   5.8   1.2
 5. static-94-32-135-161.clienti.tiscali.it     0.0%   110    8.6   6.1   3.6   9.1   1.3
 6. 74.125.49.170                               0.0%   110    3.8  14.7   3.3  87.4  15.9
 7. 108.170.245.65                              0.0%   110    5.0   4.8   4.1   6.5   0.4
 8. 216.239.50.247                              0.0%   110    3.2   3.7   3.2   4.3   0.3
 9. mil04s23-in-f110.1e100.net                  0.0%   110    4.2   3.6   3.1   4.3   0.3
 1dot1dot1dot1.cloudflare-dns.com (of course 1.1.1.1 DNS)

                                                 Packets               Pings
 Host                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. openwrt.lan                                86.9%   238    0.6   0.6   0.5   0.7   0.1
 2. static-213-205-53-217.clienti.tiscali.it    0.0%   238    2.0   2.2   1.4   4.8   0.6
 3. 94.32.155.137                               0.0%   238    1.6   2.0   1.4   2.7   0.3
 4. 94.32.154.1                                 0.0%   238    5.3   3.7   1.4   6.0   1.2
 5. static-94-32-126-25.clienti.tiscali.it      0.0%   238   10.7  12.1  10.0  14.6   1.2
 6. static-94-32-126-18.clienti.tiscali.it      0.0%   238   10.4  12.2   9.9  14.4   1.3
 7. cloudflare-nap.namex.it                     0.0%   238   10.8  10.5  10.1  11.2   0.3
 8. 1dot1dot1dot1.cloudflare-dns.com            0.0%   238   10.6  10.5  10.0  11.1   0.3
 matteo.manfredi.io (on CF)

                                                 Packets               Pings
 Host                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. openwrt.lan                                85.2%   143    0.6   0.6   0.5   0.9   0.1
 2. static-213-205-53-217.clienti.tiscali.it    0.0%   143    2.1   2.0   1.4   4.3   0.4
 3. 94.32.155.137                               0.0%   143    1.6   1.9   1.4   2.9   0.3
 4. 94.32.154.1                                 0.0%   143    5.2   3.8   1.2   6.0   1.3
 5. static-94-32-126-25.clienti.tiscali.it      0.0%   143   12.2  12.3  10.0  15.0   1.2
 6. static-94-32-126-18.clienti.tiscali.it      0.0%   143   10.2  12.2  10.0  14.2   1.2
 7. cloudflare-nap.namex.it                     0.0%   143   10.1  10.5   9.9  12.5   0.4
 8. 104.24.1.52                                 0.0%   142   10.4  10.5  10.0  11.1   0.3

Fastweb Italy not resolving to the closet datacenter
#8

Any update @cscharff?


#9

I’m also experiencing this issue of higher latency to Google services while on 1.1.1.1 (USA)

Tried many domains like google.com, www.google.com, drive.google.com, store.google.com, mail.google.com, etc. All exhibit the same results.

dfw01 POP

~10ms ping to google domains on 8.8.8.8
~10ms ping to google domains on 208.67.222.222
~45ms ping to google domains on 1.1.1.1

On 8.8.8.8:

55%20PM

On 1.1.1.1:

22%20PM

On 8.8.8.8:

37%20PM

On 1.1.1.1:

52%20PM

Seems to be routing to IAD (Virginia) instead of DFW (Texas) while using 1.1.1.1.


#10

Thanks, we’ve shared information with Google and few other major providers on the scope of our network. Since we don’t send the original client subnet to the resolvers as part of the privacy aspect of the 1.1.1.1 resolver the results can be somewhat inconsistent as Google and others work with our network team to map our infrastructure to they edns-client-subnet mappings.


#11

What about my problem with Tiscali routing to FCO instead of MXP?


#12

I’ve passed it along to the network team. Will check the status of that ticket leter today.


#13

I would contact Tiscali directly, but something tells me that I won’t get a reply. The big problem here is not the 10ms instead of 3 of latency, but the fact that FCO is much less connected to the rest of Europe, resulting in slower speeds. Would be good if you are in the south part of Italy, but when you are in the northern part…


#14

Hi @matteo,

The issue is with Tiscali’s route selection, since we use anycast to route traffic it relies on them selecting the closest route to us. I’ve contacted the Tiscali NOC to ask them to investigate why this is happening.

Thanks for the report!


#15

Hi @marty1!

Thanks a lot! Let’s hope they do not ignore the request like they would if it was me writing them! Would you mind keeping me informed if they do reply you (like they should)?

To solve the issue easily you could just deploy at TOP-IX (here in Turin), where they peer :wink:! It probably doesn’t make sense though from your network’s perspective!


#16

For sure, as for TOP-IX, this won’t help in this situation as their routers are selecting NaMeX for the whole network, adding an adjacency elsewhere won’t necessarily make them properly balance. Once they fix the issue the traffic should balance between NaMeX and MIX-IT based on user location.


#17

I assumed that was the cause of the issue, my reference to TOP-IX was just to see the reaction to mentioning it. I know you are preparing (or deploying) to DE-CIX Palermo.

In TOP-IX there is theoretically a Google Global Cache Node, don’t see it being used though anywhere, which means that there probably is a reason to deploy here. TOP-IX is just a mere 1 ms away from me, but almost nothing ends there… in the traceroute here there are even an additional two hops from the entry at TOP-IX.

                                                 Packets               Pings
 Host                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.21.0.1                                  0.0%   725    0.7   0.6   0.6   0.7   0.0
 2. static-213-205-53-217.clienti.tiscali.it    0.0%   725    2.0   1.9   1.3   6.4   0.5
 3. 94.32.155.137                               0.0%   725    2.0   1.8   1.4   3.6   0.3
 4. 94.32.154.1                                 0.0%   724    1.8   3.5   1.1   8.1   1.2
 5. itgate-pub.topix.it                         0.0%   724    1.9   2.1   1.5  12.1   0.7
 6. po2.rumba-monster.edge.trn.itgate.net       0.0%   724    2.0   4.7   1.2 194.5  17.5
 7. speed.itgate.net                            0.0%   724    1.4   1.6   1.2   6.7   0.4

#18

Would you mind verifying the usage on the deployment at FCO around this time I am writing now? I have problems maintaining a connection of a few Mbps (if you need IP and other info I can send them to you via email!) and I don’t know if the issue is from FRA (origin) to FCO or from FCO to me in Turin. I know the server can sustain a few Mbps (only user on DigitalOcean VPS).


#19

@matteo Tiscali have now fixed their routing, I can see Northern Italy going to Milano and Southern Italy going to Roma.


#20

Let me check, even though that would be superfluous I belive! That tool is cool though!

EDIT: will get back to you as soon as possible!