What is the name of the domain?
dietpi.com
What is the issue you’re encountering
Latencies and download rates are often very bad between DTAG customers and websites behind Cloudflare
What steps have you taken to resolve the issue?
Contact with Deutsche Telekom AG as ISP, in attempt to push the topic their end, without success so far.
What are the steps to reproduce the issue?
From an Internet access point with Deutsche Telekom AG (Telekom in Germany or one of their daughter providers in other European countries) connect to or download from websites behind the Cloudflare CDN (free plan). Depending on time and day, latencies can be very long, 2 seconds and more, and download rates can go down to 50 KiB/s.
Traceroutes (MTR) often show routing over USA/New York, while Cloudflare runs a lot of Edge server in Germany and Europe:
<host> (<IPv6>) -> dietpi.com (2a06:98c1:3121::3) 2024-07-04T16:20:46+0200
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. p200300e06f1e54c55a687afffe1b2a8c.dip0.t-ipconnect.de 0.0% 21 1.7 2.1 1.7 2.5 0.2
2. 2003:0:8703:9000::1 0.0% 21 11.4 12.8 10.6 42.5 6.8
3. 2003:0:f00::782 65.0% 21 112.3 126.3 112.3 202.4 33.6
4. 2003:0:f00::783 0.0% 21 112.6 112.9 112.3 115.0 0.7
5. if-ae-2-2.tcore2.n0v-newyork.ipv6.as6453.net 0.0% 21 111.5 111.4 111.0 112.2 0.3
6. if-ae-23-2.tcore4.njy-newark.ipv6.as6453.net 0.0% 20 113.5 113.8 113.3 115.0 0.4
7. if-ae-1-3.tcore3.njy-newark.ipv6.as6453.net 0.0% 20 115.9 115.3 114.6 115.9 0.3
8. 2001:5a0:f00::1e 5.0% 20 119.0 127.6 118.7 161.0 9.9
9. 2400:cb00:452:3:: 0.0% 20 122.0 130.9 119.0 150.9 11.9
10. 2a06:98c1:3121::3 0.0% 20 128.1 124.6 118.5 131.3 4.2
<host> (<IPv4>) -> dietpi.com (188.114.96.3) 2024-07-04T16:22:12+0200
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. speedport.ip 0.0% 24 2.2 1.8 1.2 2.2 0.3
2. p3e9bf7f2.dip0.t-ipconnect.de 0.0% 24 10.8 11.6 10.7 14.4 0.9
3. nyc-sb6-i.NYC.US.NET.DTAG.DE 0.0% 24 106.6 107.2 106.6 108.8 0.4
4. nyc-sb6-i.NYC.US.NET.DTAG.DE 0.0% 24 106.4 108.2 106.2 146.6 8.2
5. 80.156.160.213 0.0% 24 104.5 104.8 104.3 106.6 0.4
6. if-ae-0-2.tcore3.njy-newark.as6453.net 0.0% 24 116.1 116.0 114.7 125.1 2.2
7. 66.198.70.2 12.5% 24 111.7 115.4 111.3 125.2 4.0
8. 162.158.61.221 0.0% 24 107.8 108.1 106.2 115.9 2.3
9. 188.114.96.3 0.0% 23 111.6 111.0 110.0 116.1 1.3
Download rates are okay currently, usually become very bad in the evening and on weekends. It shows however the unnecessary routes over USA/NY, using IPv6 as well as IPv4. I can add further data later, showing worse latencies and download rates.
After discussion on Telekom community forum and investigation, it seems there is no peering contract between Cloudflare and DTAG. Hence packets are routed through intermediate providers, which can include such located in the USA. In this case (not always), it is even handed over form a DTAG node located in the USA. There are however other examples with intermediate nodes all in Germany, and still long latencies and bad transfer rates.
For peering, ISPs usually expect payment from other service providers. Already 10 years ago, Level 3 (backbone provider e.g. serving Twitch streams that time) was criticising ISPs like Verizon and DTAG for expecting way above common market prices: Netzneutralität: Backbone-Betreiber Level 3 äußert sich zu Peering-Problemen | heise online
It might be still the same today, leading to this problem. Other (known) providers in Germany do not have this problem.
We are trying to drive the topic on DTAG end, but I also want to drive it here, also to collect info for other Cloudflare customers which might be notified some of their visitors about bad connectivity. At least some information about whether my above assumptions are true, and in case about the state of negotiations (as much as it can be made public) would be great, so we know what to expect in shot and long term, and can decide what to do. Having each sides just blaming the other one, does not help us customers and visitors.
More reports and information can be found on the sadly closed topic: Latency to proxied DNS entries high since 2024-01-30T07:09:15CET
And here one thread discussing it at the German DTAG community forum: https://telekomhilft.telekom.de/t5/Festnetz-Internet/Peering-Routing-zu-Cloudflare-immer-wieder-extrem-schlecht/td-p/6766121
In the “accepted answer” (not solving anything), the community manager mentioned that “despite peering, packets are going a way not designed for this” and that they are “not able to solve some branches without cooperation with Cloudflare”. The way I understand the routing with intermediate providers, there is no direkt peering. However, it is obvious that both sides have to communicate and in case negotiate to solve this. I hope there is some motivate for Cloudflare end to enable stable latencies and download rates for visitors for a major ISP in Germany (the largest one in Germany) and all Europe.