Latency to proxied DNS entries high since 2024-01-30T07:09:15CET

Hi!

I’m using the DNS Proxy feature to route traffic to specific targets through Cloudflare.

Since 2024-01-30 approx 07:09:15CET I’m seeing very high latency to the servers as you can see on the following graph:

The same ICMP requests are also sent to Google and other targets and those latencies did not change.

I’m located near to Frankfurt in Germany. However, looking at a tracepath to the IP resolved by DNS, I see that traffic is routed through to nyc-sb5-i.NYC.US.NET.DTAG.DE which indicates that the requests are served from a US datacenter.

Am I the only one seeing this problem? There is another post regarding slow pages when using Cloudflare: https://community.cloudflare.com/t/page-slow-when-using-cloudflare-fast-when-accessing-directly-by-ip/608078/6

2 Likes

Looks like my ISP is at fault here.
IIRC Cloudflare announces the same IPs (e.g. for the DNS record) on all their POPs and relies on routing algorithms to find the closest destination.

As the US hop provided in my first post seems to still belong to my ISP, it’s their internal routing doing weird things…

1 Like

Exact same issue here. Super frustrating. Have you been in touch with them (Telekom)? Unfortunately my German is really bad, so me reaching out to them isn’t exactly feasible.

The problem is that I’m not a direct customer of DTAG but customer of a sub-contractor who uses their infrastructure and backbone.
I haven’t tried reaching out to them yet. But I guess it’ll be pointless…

As a DTAG customer I had the exact same issue here with a dev site (Cloudflare Pages / paid plan). At the end of the day I switched from CF Free to Pro plan. A few minutes later, my site had a new IP address and the pings were back in the 10ms range.

From other providers (Azure, AWS, Google, Hetzner), the pings in the Free Plan were also in the single-digit ms range before. This may have something to do with the fact that DTAG and CF have been arguing about peering in DE for years. But the reason may also be this strange “TCP Turbo” option, which is not available in the Free plan.

Since today 17:29:30CET 1.1.1.1 is also affected by the same problem. The latency is slightly less (about 20ms lower) but increased by about 12x compared to before. Also package loss seems to have increased a lot in the last days.

I’m a DTAG customer and have observed the same issue.

My traffic to certain Cloudflare sites is slow as ■■■■, sometimes taking 3 minutes to load 200kb, other times packets are lost or delivered twice several minutes later, resulting in double posts in chat apps.

I have several sites behind Cloudflare and only the ones that take the route via nyc-sb5-i.NYC.US.NET.DTAG.DE are having these problems.

Funnily enough, it also started 6 days ago for me on Tuesday morning, same as you.

1 Like

I just stumbled across this accidentally. We’ve also noticed very degraded performance from the DTAG network (AS3320) to mulitple sites which are proxied through Cloudflare. I’ve opened a ticket about it but have not received a response yet.

That Free and Paid plans have that vast different routings is new to me but I can confirm that my sites on Free go to EWR and on Pro to FRA.
I really hope that this is some kind of emergency solution that was deployed and not something that will stay.

As a DTAG customer myself I can confirm that, especially in the evening hours, access to many Cloudflare-proxied sites is nearly impossible with 1+ minute load times and up to 50-60% packet loss.

I was wondering if i just configured something wrong or so, but no, i got the same issue as you all got with my DTAG connection. I got access to two domains that use cloudflare proxy. One has it’s origin Server located in Düsseldorf(Germany) and the other one in Frankfurt(Germany).

Only the Domain with it’s origin in Frankfurt has the issue that it routes over the following DTAG host in NYC: nyc-sb5-i.nyc.us.net.dtag.de

Because this is all in the DTAG network, i guess they got the issue, but it could also be an issue of Cloudflare i guess. No matter who is at fault, i really hope this get’s fixed asap.

regards,
Manuel

just tested something else, if i proxy the same server over the other domain on cloudflare it routes over frankfurt as expected.

So only a few of cloudflares Proxy servers/IPs seem to be affected.

I my case 104.26.13.219 routes correctly, 172.67.190.223 goes over NYC for no reason.

Is one of those domains on a pro plan? For me it doesn’t matter where the server is located, all that matters is whether the domain is on a free or paid plan.

yeah, did some more research yesterday, as mentioned by others in some other topics CF is routing over NYC on free plans due to issues with DTAG wanting (i guess) way to much money for peering. Thanks DTAG for degrading the experience for a lot of germans just to get even richer…

In the meantime I got a response from the CF support about this issue:

“We are aware of the issue and our network engineers are working on a solution with Deutsche Telekom, but we do not have an ETA on when this will be solved, as it needs to be solved from Deutsche Telekom side.”

5 Likes

I habe the same massive problem!

See my post:

Actually, I should cancel my T-Online / Telekom Internet access and telephone connection (Business) right now and switch to Vodafon.

They wouldn’t have deserved anything else at DTAG.

You can do that as a residential customer but there is no guarantee that Vodafone (which is also a s**t company) and Cloudflare won’t be having a dispute with each other in the future. All of these are for profit megacorporations and they don’t care about us as long as there are no consequences. The only thing they care about is how much they can profit off of us to make their shareholders happy. We need regulation. The EU needs to step up and fine them for restricting access to basic services.

I am in Germany and a Telekom customer. So I pay these people an expensive monthly fee so that I can “view” my own website slowly as it is redirected thousands of kilometres. Out of pure “shareholder value”.

It’s long past time to massively limit the power of large corporations.

1 Like

We have the same problem with two Deutsche Telekom ISP customers, being routed from a German Deutsche Telekom node to a New York (USA) Deutsche Telekom node, and from there back to our German VPS, resulting in ~1-2 seconds response times:

traceroute: Warning: dietpi.com has multiple addresses; using 188.114.97.3
traceroute to dietpi.com (188.114.97.3), 64 hops max, 52 byte packets
1 fritz.box (192.168.1.1) 3.911 ms 2.343 ms 5.686 ms
2 p3e9bf6a8.dip0.t-ipconnect.de (62.155.246.168) 14.931 ms 10.896 ms 17.432 ms
3 nyc-sb5-i.nyc.us.net.dtag.de (62.154.5.157) 100.103 ms 100.446 ms 100.112 ms
4 87.128.239.250 (87.128.239.250) 109.588 ms 115.406 ms 116.668 ms
5 if-ae-0-2.tcore3.njy-newark.as6453.net (216.6.90.14) 115.654 ms * *
6 * 66.198.70.2 (66.198.70.2) 107.806 ms 111.245 ms
7 172.70.228.2 (172.70.228.2) 141.449 ms
199.27.132.36 (199.27.132.36) 104.642 ms *
8 188.114.97.3 (188.114.97.3) 105.747 ms 107.653 ms *
tracert dietpi.com

Routenverfolgung zu dietpi.com [104.21.28.141]
über maximal 30 Hops:

  1     2 ms     4 ms     2 ms  fritz.box [192.168.0.1]
  2     4 ms     3 ms     4 ms  speedport.ip [192.168.2.1]
  3    10 ms     9 ms    24 ms  p3e9bf094.dip0.t-ipconnect.de [62.155.240.148]
  4   141 ms   111 ms   210 ms  nyc-sb5-i.NYC.US.NET.DTAG.DE [62.154.5.161]
  5   109 ms   108 ms   108 ms  87.128.239.250
  6   108 ms   108 ms   110 ms  if-ae-0-2.tcore3.njy-newark.as6453.net [216.6.90.14]
  7   114 ms     *      119 ms  66.198.70.2
  8   114 ms   133 ms   137 ms  172.70.228.2
  9     *      120 ms   108 ms  104.21.28.141

Ablaufverfolgung beendet.

Since the route via USA is done between two Deutsche Telekom nodes, I guess it is more a problem with Telekom, not so much with Cloudflare. Also other German provides do not have this issue. But not sure where Telekom gets the routing information from, probably Cloudflare is indirectly involved as well?

This only happens on the free Cloudflare tier. Any sites on the Business tier or above does not suffer from any of the issues mentioned here, such as increased latency and packet loss.

Btw the related discussion at Deutsche Telekom community: Gelöst: Oft sehr langsame Verbindung. wenn Website hinter ... | Telekom hilft Community

@WasserEsser Interesting, are packets from Deutsche Telekom not routed over USA when you use the Business tier? I wonder how Cloudflare can even have an effect on this, as this all happens within Deutsche Telekom network.

Even the Pro plan routes the traffic just fine.
They separate subnets between plans. Either there is some BGP magic or DTAG and Cloudflare have some agreement for certain subnets.

From my point of view:
Pro plan site: DTAG DE → Level3 in Frankfurt → Cloudflare in Frankfurt
Free plan site: DTAG DE → DTAG New York → TATA New York → Cloudflare New York

2 Likes