Our server in Frankfurt (Germany, EU) is being proxied from the Newark, NJ (US) datacenter

So here is our problem:

We have a website with cloudflare proxy enabled in the dns settings. The server is located in Frankfurt (Germany) and our visitors are from the EU (our website’s language is hungarian).

Up until about 2 weeks ago the website’s assigned cloudflare datacenter was Vienna (Austria, EU): colo=VIE

Then for some reason unknown to us it was assigned to the Newark datacenter (NJ, USA): colo=EWR

This change in the datacenter location instantly dropped the website’s performance to unacceptable levels for EU visitors, who make up 99% of our target audience.

We’ve waited for cloudflare to detect and mitigate this problem on their own but unfortunately it didn’t happen, so this is why I’m trying to reach support here. We don’t have a pro subscription so we can’t influence the datacenter location, and it seems our only option for mitigation is to completely disable cloudflare proxy for the domain, but then of course we’ll have no ddos protection and no caching. Obviously we don’t want to go ahead with this, but if we can’t make sure that our website gets served from the same continent where our server is hosted and our visitors are located, then we’re going to have to disable proxying.

If anyone from cloudflare support can read this, please advise how to proceed. We’ve used this service in the past years without any problem and we’d prefer to continue to do so, but this is forcing us off board.

Thank you in advance!

2 Likes

Cloudflare serves every website from (almost) every data center.

Which data center you actually use is (usually) not decided by Cloudflare, but by your ISP.

Can you share your domain so I can check which data center I connect to from Germany?

1 Like

Hi Laudian, thanks for the reply! This is the link: https://cdn.444hsz.com/cdn-cgi/trace

1 Like

I understand that the node cloudflare chooses depends on the user’s location. Like most of our users I’m located in Hungary and my isp is Magyar Telekom (https://telekom.hu), which is a subsidiary of Deutsche Telekom.

1 Like

Cloudflare doesn’t choose a location, your ISP does. From home (ISP: Deutsche Telekom), I also see colo=EWR.
From my server (via Hetzner), I see colo=AMS (Amsterdam)

I’d bet that you see other locations as well if you test from a non-Telekom ISP.

2 Likes

Unfortunately this is the case for all websites behind a Free plan if the visitor is using Magyar Telekom (I didn’t know yet that Deutsche Telekom is also affected though), I also made a post about it a few days ago.

1 Like

Ok, I went thru this thread and the conclusion is off-putting, to say the least: https://telekomhilft.telekom.de/t5/Festnetz-Internet/3-Tage-Cloudflare-Probleme/td-p/6313070

Here is a summary of my findings:

Questions:

  1. First and foremost can we get an official answer to this: Is a paid plan required now to properly service ALL users in the EU including customers of DTAG? If so, which plan do we need?

  2. Is it possible for Cloudflare to provide us additional information regarding the dispute with DTAG? If there is non-confidential information available, it should be made public, so customers can put pressure on DTAG, and in case they want to file a complaint with the EU regulatory body to get them to investigate a possible breach of consumer rights.

  3. As Cloudflare basically have a monopoly in providing reverse proxy/ddos protection services free of charge for websites, does it consider itself a gatekeeper to a huge portion of the internet?

Looking forward to getting official answers if possible. Thank you in advance!

6 Likes

You are unlikely to get an official response here, this blog will cover it…

My guess (I am just a user, not staff, and it is my personal guess only), from what is happening with DTAG and seemingly in India, is that Cloudflare has had to relent and pay for some peering to these networks as the telcos try to flex their muscle. That likely comes at great cost so is only implemented for paid or higher level plans. That DTAG now seems to be using such crazy long routing would appear (no evidence, just personal opinion) to be an escalation.

Arguments are likely:

  • Cloudflare wants free and widespread peering to benefit internet users (and, of course, itself)
  • Telcos want a slice of the pie (for shareholders, for investment, etc) and want to charge heavy traffic routes far more

For DTAG, your best option is to complain to the German regulator in the short term and to lobby your EU representative as in the blog post to legislate.

4 Likes

Regarding your question #1:
We switched from CF Free to the Pro plan because of this problem. Within a few minutes we got new IPs and our sites were delivered from CF EU POPs again.

See here…

4 Likes

I’ve also posted in my thread over at Latency to proxied DNS entries high since 2024-01-30T07:09:15CET - #5 by user3383, but just to also mention it here: since today 17:29:30CET 1.1.1.1 is also affected by the same problem. The latency is slightly less (about 20ms lower) but increased by about 12x compared to before. Also package loss seems to have increased a lot in the last days

3 Likes

I would also appreciate an official communication by CF on this issue.

As both a DTAG customer and user of cloudflare workers / pages / R2, this is extremely frustrating as it makes Cloudflare Free pretty much unviable as a hosting option.

Having the Pro plan apparently solve this is good to know, but $25/month is too steep a price for most hobby projects and it would also be a lot cheaper to just move the projects somewhere else for the amount of usage these kinds of projects usually get.

1 Like

Thanks the replies guys. We are evaluating other alternatives atm. We already pay 100+ eur/mo for our simple website, and adding another $25/mo (or $240/y if paid upfront) subscription is not something we want to do. And not only that. Cloudflare’s complete lack of transparency regarding this issue is worrying. They don’t offer any guarantee that the Pro plan will ensure access to the “fast lane”. In fact if most of the free plan users subscribe and cause congestion again in their peering, they can just switch the eligibility to a higher tier silently, just like they did before.

Apparently someone in another thread has received a reply from cloudflare, linking here for visibility: Latency to proxied DNS entries high since 2024-01-30T07:09:15CET - #13 by kevin104

2 Likes

I have the same problem and I hope we have a solution soon from CF and EU very soon!

1 Like

Received a reply from the german regulator stating that peering agreements are not something they regulate and not something that falls under the current net neutrality laws. The full german answer is below:

Das (Routing-)Problem scheint die Zusammenschaltung im Internet zu betreffen. Oftmals tauschen Netze im Internet ihren Verkehr wechselseitig aus (Netz A transportiert den Verkehr von Netz B und umgekehrt), ohne dass sich die Netzbetreiber dies gegenseitig in Rechnung stellen; dies wird als „settlement free peering“ bezeichnet. In den Foren im Internet wird dann oftmals vermutet, der Internetzugangsanbieter Peering-Kapazitäten schlecht nutzen oder dass sogar eine Drosselung von Inhalten stattfinde.

Allerdings unterliegt die Zusammenschaltung im Internet grundsätzlich nicht der Regulierung. Ob ein Unternehmen mit einem Anderen eine Peering-Vereinbarung eingeht - bzw. zu welchen Konditionen – ist somit Sache der beteiligten Unternehmen.

BEREC – das Gremium europäischer Regulierungsstellen – hat in Ihren bisherigen Berichten festgestellt, dass sich der IP-Zusammenschaltungs-Markt insgesamt gut und ohne signifikante regulatorische Interventionen entwickelt hat. Es liegen auch keine entsprechenden Beschwerden von Anbietern bei der Bundesnetzagentur vor.

Die Zusammenschaltung im Internet liegt nicht im Anwendungsbereich der Netzneutralitätsverordnung. Damit stellen individuelle Vereinbarungen über Kapazitäten keinen Verstoß gemäß der „Netzneutralitäts-Verordnung“ dar. Netzneutralität liegt also vor, wenn der Verkehr beim Transport innerhalb der Infrastruktur des ISP gleich (nichtdiskriminierend) behandelt wird. Die Bereitstellung und Dimensionierung von Kapazitäten im Rahmen der IP-Zusammenschaltung ist nicht Bestandteil der „Behandlung des Verkehrs bei der Bereitstellung von Internetzugangsdiensten“.

2 Likes

In my opinion, this isn’t really an issue for the Bundesnetzagnetur, but for the Bundeskartellamt (Antitrust).

Telekom is using their power in one market (ISP) to enforce their position in the Transit market.

Bundeskartellamt - Missbrauchsaufsicht.

But there is absolutely no way the Bundeskartellamt would ever move against Telekom.

4 Likes