Anycast Issue

I process data packets from CERN which use Cloudflare’s CDN *.openhtc.io.
Since I’m located in Germany the downloads are usually routed as follows:
CERN (Geneva) → Cloudflare Datacenter in Europe (mostly FRA, AMS or LHR) → my location

For a couple of days Cloudflare routes the downloads as follows:
CERN (Geneva) → Cloudflare Datacenter EWR (Newark, US) → my location

That way the downloads have to cross the Atlantic twice which causes significant delays.
A colleague located in the Netherlands gets his downloads via AMS (as expected).

Can you elaborate on where you see these issues?

  1. What ISP / provider, preferably their AS number?
    The AS number can be found here:
  1. Country is Germany, … but what state/region?

This information (especially the AS number) would be essential for Cloudflare (and/or any other network) in order to be able to dig further in to such issues.

Where you are being routed to, depends on your ISP’s decisions, including their peering arrangements with Cloudflare, and whether these peering arrangements (if they have such) are done locally or far away.

If your provider from Germany does not want to peer with Cloudflare within Germany, and is only peering with Cloudflare in e.g. Newark, New Jersey, US, then traffic from that ISP will be sent from Germany and to Cloudflare’s PoP in Newark, New Jersey, US.

Thanks.

As for the requested information:

https://one.one.one.one/help/#eyJpc0NmIjoiTm8iLCJpc0RvdCI6Ik5vIiwiaXNEb2giOiJObyIsInJlc29sdmVySXAtMS4xLjEuMSI6IlllcyIsInJlc29sdmVySXAtMS4wLjAuMSI6IlllcyIsInJlc29sdmVySXAtMjYwNjo0NzAwOjQ3MDA6OjExMTEiOiJObyIsInJlc29sdmVySXAtMjYwNjo0NzAwOjQ3MDA6OjEwMDEiOiJObyIsImRhdGFjZW50ZXJMb2NhdGlvbiI6Ik1VQyIsImlzV2FycCI6Ik5vIiwiaXNwTmFtZSI6Ikdvb2dsZSBTZXJ2ZXJzIiwiaXNwQXNuIjoiMTUxNjkifQ==

You are connecting from:
Deutsche Telekom AG (AS3320)

https://bgp.he.net/
Your ISP is AS3320 (Deutsche Telekom AG)

A bit north of NUE.
Roughly in the middle between FRA and MUC.

I was suspecting this when you mentioned Germany, I’ll therefore quote what I wrote in another thread recently:

Unless AS3320 Deutsche Telekom wants to man up and participate in creating a better Internet, I’m unfortunately afraid you shouldn’t be expecting any remediation from AS3320 Deutsche Telekom (or any of it’s subsidiaries).

AS3320 Deutsche Telekom’s bad decisions is not only hitting Cloudflare, but also (many) other networks.

1 Like

For a better understanding on my side.

When my systems make a request to *.openhtc.io they first resolve the IP address.
This is what Cloudflare replies:

dig @1.1.1.1 openhtc.io.

; <<>> DiG 9.18.24 <<>> @1.1.1.1 openhtc.io.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63816
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openhtc.io.                    IN      A

;; ANSWER SECTION:
openhtc.io.             300     IN      A       172.67.158.11
openhtc.io.             300     IN      A       104.21.49.17

;; Query time: 19 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Wed May 01 17:03:23 CEST 2024
;; MSG SIZE  rcvd: 71

Cloudflare’s DNS replies from the datacenter in MUC (excerpt from traceroute):

7  cloudflare-gw.cr0-muc1.ip4.gtt.net (141.136.100.98)  32.810 ms  32.356 ms  32.143 ms
8  one.one.one.one (1.1.1.1)  21.587 ms  15.835 ms  15.952 ms

Since openhtc.io is also a CLoudflare service I would expect the DNS to reply an IP pointing to a European datacenter.
That’s how it worked in the past. I mostly got FRA, sometimes AMS or LHR, in rare cases MIL, but all from Europe and Deutsche Telekom always set a correct route.

Now DNS resolves an IP from the US and Deutsche Telekom routes me there.
From their point of view this is correct, isn’t it?

BTW:
Other Cloudflare services like speedtest stay within Germany and use a route to the datacenter in MUC.

Sure, don’t hesitate to ask if there is anything you’re wondering about. :slight_smile:

Correct.

From my end, that seems correct as well.

These two IP addresses are the ones you need to check against.

1.1.1.1 (or any other IP address announced by Cloudflare) has no relation, and can give completely different results.

It wouldn’t be technically impossible either, that 1.1.1.1, 104.21.49.17, and 172.67.158.11, even if they all lead to Cloudflare, could be taking three completely different paths before ending up on Cloudflare.

Things can easily be misunderstood, - so let me clarify:

OpenHTC is NOT a Cloudflare service.

That being said, the owner of OpenHTC do use Cloudflare services, in front of their website.

From Denmark, I am reaching Cloudflare Copenhagen (CPH) with all the three IP addresses, 1.1.1.1, 104.21.49.17, and 172.67.158.11 mentioned above.

From United Kingdom, I am reaching Cloudflare London (LHR) with all the three IP addresses, 1.1.1.1, 104.21.49.17, and 172.67.158.11 mentioned above.

So, that leads me towards the question, … what do you call a European datacenter? :thinking:

When in the past would that be?

And what exactly did you do to check this, how often did you do that check, and so on?

Both and.

For the former part, you can argue that 104.21.49.17, and 172.67.158.11 are US IP addresses, as the two IP addresses are allocated from the American Registry of Internet Numbers (ARIN), to Cloudflare.

However, the 1.1.1.1 is currently held by Asia Pacific Network Information Centre (APNIC), with the description:

descr:          APNIC and Cloudflare DNS Resolver project
descr:          Routed globally by AS13335/Cloudflare
descr:          Research prefix for APNIC Labs
country:        AU

Forgive me for asking, but with that logic:

Shouldn’t 1.1.1.1 take you to Asia, or at least Australia? :thinking:

When you say “Other Cloudflare services”, I will again point towards what I mentioned above, that OpenHTC is NOT a Cloudflare service.

Cloudflare is sending out all their IP addresses, including 1.1.1.1, 104.21.49.17, and 172.67.158.11 from several hundred datacenters across the whole world, using anycast routing, as you mentioned in your thread title.

From multiple locations across the European continent, as explained above, I am getting to a local datacenter, in the same country as the test is conducted from, with the mentioned IP addresses.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.