Cable Bahamas - very slow traffic from CDNs

DNS & Network 1.1.1.1
, ,
#1

Output from https://Cloudflare-dns.com/help

I have noticed that since switching to Cloudflare’s 1.1.1.1 service a number of CDN-backed networks - notably Apple’s software delivery - are extraordinarily slow (<50Kbps), leading to, for example, the macOS Mojave 10.4.2 update and the latest iOS updates taking many hours to download.

I also have the same issue with many of the statically hosted files on GitHub, which appear to be hosted on AWS S3. Google’s 8.8.8.8 service appears to give different IP addresses for the same endpoints:

nslookup github-production-release-asset-2e65be.s3.amazonaws.com 
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
github-production-release-asset-2e65be.s3.amazonaws.com canonical name = s3-1-w.amazonaws.com.
Name: s3-1-w.amazonaws.com
Address: 52.216.132.179 

Server: 1.1.1.1
Address: 1.1.1.1#53
Non-authoritative answer:
github-production-release-asset-2e65be.s3.amazonaws.com canonical name = s3-1-w.amazonaws.com.
Name: s3-1-w.amazonaws.com
Address: 52.217.0.99

Admittedly this could be more to do with Amazon’s Route53 service and unfortunately traceroute to either of these IP addresses returns nothing beyond my local ISP’s network, all ICMP traffic is then dropped on every endpoint.

Overall speed is okay - measuring with speedtest dot net for example tracks >40Mpbs consistently.

dig shows the location being used as MIA - which I’m assuming is Miami, which would be optimal (unless of course you want to host something here in the Bahamas…)

Responses overall are pretty fast and 1.1.1.1 does seem extremely performant; however Apple updates are unfortunately a large part of my family’s network traffic and Cloudflare does seem to be routing it sub-optimally to the upstream CDN.

traceroute responses:
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
1 localrouter.localdomain (192.168.1.1) 9.398 ms 3.356 ms 2.121 ms
2 10.22.0.1 (10.22.0.1) 10.892 ms 15.069 ms 11.697 ms
3 10.254.1.177 (10.254.1.177) 14.329 ms 18.573 ms 16.859 ms
4 10.254.0.13 (10.254.0.13) 12.059 ms 11.286 ms 14.707 ms
5 10.254.0.2 (10.254.0.2) 18.107 ms 26.885 ms 18.180 ms
6 xe-5-1-9.0-brx-teracore02.cwc.com (63.245.3.137) 49.497 ms 48.718 ms 49.768 ms
7 * * *
8 one.one.one.one (1.1.1.1) 25.360 ms 18.115 ms 19.721 ms

traceroute to 1.0.0.1 (1.0.0.1), 64 hops max, 52 byte packets
 1  localrouter.localdomain (192.168.1.1)  8.048 ms  2.585 ms  1.821 ms
 2  10.22.0.1 (10.22.0.1)  13.020 ms  11.224 ms  12.222 ms
 3  10.254.1.177 (10.254.1.177)  13.425 ms  15.119 ms  11.732 ms
 4  10.254.0.13 (10.254.0.13)  12.425 ms  9.832 ms  12.276 ms
 5  10.254.0.2 (10.254.0.2)  20.113 ms  19.565 ms  17.304 ms
 6  xe-5-1-9.0-brx-teracore02.cwc.com (63.245.3.137)  50.564 ms  50.164 ms  52.036 ms
 7  * * *
 8  one.one.one.one (1.0.0.1)  27.279 ms  23.587 ms  20.758 ms

dig
; <<>> DiG 9.10.6 <<>> +tcp @1.1.1.1 id.server CH TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63604
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;id.server. CH TXT
;; ANSWER SECTION:
id.server. 0 CH TXT “MIA”
;; Query time: 22 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sat Dec 08 15:44:07 EST 2018
;; MSG SIZE rcvd: 54

; &lt;&lt;&gt;&gt; DiG 9.10.6 &lt;&lt;&gt;&gt; +tcp @1.0.0.1 id.server CH TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 48089
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;id.server. CH TXT
;; ANSWER SECTION:
id.server. 0 CH TXT "MIA"
;; Query time: 27 msec
;; SERVER: 1.0.0.1#53(1.0.0.1)
;; WHEN: Sat Dec 08 15:44:09 EST 2018
;; MSG SIZE rcvd: 54
#2

Throughput to servers is not DNS related.

The most likely reason is, what you already discovered, that you are served a different IP address. For privacy reasons Cloudflare does not provide the client IP to the autoritative resolver (https://developers.cloudflare.com/1.1.1.1/nitty-gritty-details/), hence it wont return an IP closest to you but a generic one, which presumably provides a lower throughput.

1 Like
#3

Hi Sandro, thanks for replying and for clarifying the behaviour of 1.1.1.1 and highlighting that it does not send the eDNS client subnet headers. This is almost certainly causing, or at least contributing to, the woeful performance received when accessing many CDN-backed resources. Unfortunately this behaviour will prevent me from using Cloudflare’s 1.1.1.1 service.

#4

Do you have an example of fast and slow hostnames and IP addresses, with Apple updates or other things? With what you get when using 1.1.1.1 or other DNS?

Amazon S3 isn’t a distributed CDN – it uses a large number of IPs for load balancing or other reasons, but a single bucket is located in a single area, and the different IPs should probably have equivalent performance.

github-production-release-asset-2e65be.s3.amazonaws.com is located in the us-east-1 AWS region in the Washington, DC, US, area no matter what its IP is.

#5

Certainly - the two most recent downloads I have had issues with are:

Apple: http://updates-http.cdn-apple.com/2018/macos/091-97094-20181205-10DCDA1A-DEB9-11E8-9609-7231290FBED4/macOSUpd10.14.2.dmg

GitHub: https://github.com/brave/brave-browser/releases/download/v0.57.18/Brave-Browser.dmg (but I’ve noticed every other download I have tried comes from the same S3 bucket as listed in my original post)

I have this afternoon switched to using Google’s 8.8.8.8 and throughput from these two sites is improved. Not great, which suggests issues with my ISP as well (which I am raising via a support call now) but certainly better (around 200kbps when using 8.8.8.8 vs around 50/75kbps when resolving using 1.1.1.1).

#6