Egypt DOWN: [188.114.97.6] AMS Routing Fatalities

VERY frustrating that Cloudflare keeps showing Amsterdam (AMS) as “Operational” when in real life that routing between Egypt is 100% Down.

All Egypt networks I’ve tested are unable to reach 188.114.97.6 however, routing DOES work for the nearby network 188.114.96.6. Fortunately, some web client applications are able to just choke on 188.114.97.6 for a few minutes and then eventually Cloudflare will Round Robin over to the 188.114.96.6 route allowing the client to breathe again, but the experience is very horrible.

$ nslookup ipconfig.io 1.1.1.1
Server:         1.1.1.1
Address:        1.1.1.1#53

Non-authoritative answer:
Name:   ipconfig.io
Address: 188.114.97.6
Name:   ipconfig.io
Address: 188.114.96.6

$ time curl -v ipconfig.io
* About to connect() to ipconfig.io port 80 (#0)
*   Trying 188.114.97.6... *CHOKE* *CHOKE* *CHOKE* *CHOKE* Connection timed out
*   Trying 188.114.96.6... connected
* Connected to ipconfig.io (188.114.96.6) port 80 (#0)
[...]
41.223.53.XXX
* Connection #0 to host ipconfig.io left intact
* Closing connection #0

real    1m3.467s
user    0m0.007s
sys     0m0.008s
$

The work-around is to DISABLE the bleeding orange “Proxy Status” from “Proxied” to “DNS Only”. Or tell everyone in Egypt that wants to access any “Cloudflare Proxied” site to leave the country first.

A better solution is to pray Cloudflare will QUIT resolving any domains to 188.114.97.6 until this routing malfunction has been repaired. I believe this issue is a known problem already:

Another example blocked IP is [41.223.53.1]. This has been broken for over a month and it’s still 100% packet loss today, yet that ticket was closed for some reason!

This is NOT a “Support” issue. This is an outage that Cloudflare must address. At least report “Partial Outage” for Amsterdam Netherlands data center instead of “Operational” until this is functioning again.

I understand your frustration, but I am confused as why you think Cloudflare needs to change the status when it is not their issue. Why should Cloudflare abandon an IP range that works for the world minus one country?

2 Likes

Until Cloudflare can resolve the routing issue with NTT (CF’s direct transient provider for Amsterdam) then Cloudflare must quit propagating the DNS to 188.114.97.6/31 for anyone attempting to resolve from Egypt.

And Cloudflare needs to flag AMS as a known outage until this is resolved.

Cloudflare needs to quit blocking Egypt on Cloudflare’s Amsterdam routers.

Remember that routing works for 188.114.97.5 but not for 188.114.97.6 even though both IPs are Cloudflare.

Either Cloudflare is intentionally dropping the packets to or from the Egypt networks or else Cloudflare has unintentionally misconfigured their routers accidentally. But it’s quite clear that nobody else can fix this except Cloudflare. How is this even confusing?

Any Cloudflare admin can do a traceroute from 188.114.97.6 or 188.114.97.7 to any Egypt IP and the problem will immediately exploit.

188.114.97.5 works fine
188.114.97.6 is broken
188.114.97.7 is broken
188.114.97.8 works fine

$ time mtr -rw -c50 188.114.97.5
HOST: 41.223.53.1      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 41.215.241.41  82.0%    50   13.2   6.6   0.7  15.5   6.2
  2. 196.46.189.109  0.0%    50    0.7   0.9   0.6   7.9   1.1
  3. 196.46.189.229  0.0%    50    0.8   1.1   0.6   6.2   1.1
  4. 196.46.189.33   0.0%    50    1.1   1.2   0.4  26.8   3.7
  5. 172.29.120.1   10.0%    50   69.8  63.2  61.2  74.8   3.3
  6. 172.17.51.34    4.0%    50   62.3  62.3  61.2  75.6   2.2
  7. 185.84.18.93    8.0%    50   91.8  92.3  90.6 104.2   2.4
  8. 185.84.18.70    6.0%    50   95.3 109.6  94.1 162.9  17.1
  9. 162.158.20.240  4.0%    50   94.3  96.8  93.6 121.9   6.1
 10. 188.114.97.5    8.0%    50   93.5  94.0  93.5  95.0   0.4
$ mtr -rw -c50 188.114.97.6
HOST: 41.223.53.1      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 41.215.241.41  94.0%    50    0.9   1.1   0.8   1.5   0.4
  2. 196.46.189.109  0.0%    50    0.8   1.8   0.6  15.6   2.8
  3. 196.46.189.229  0.0%    50    0.9   1.1   0.6   8.8   1.4
  4. 196.46.189.33   0.0%    50    0.9   0.8   0.3   6.5   1.0
  5. 172.29.120.1    4.0%    50   60.5  62.4  60.5  81.0   3.2
  6. 172.17.51.34    6.0%    50   61.5  62.0  61.1  75.6   2.1
  7. ???            100.0    50    0.0   0.0   0.0   0.0   0.0
$
1 Like

What makes you Cloudflare is blocking the connections?

If you read what this user got from Cloudflare support, it states that Egypt is doing the blocking.

If the routing to Cloudflare was failing or bad, then you would see the whole subnet fail and fail from everywhere.

Yes, it’s possible that Cloudflare is NOT intentionally blocking Egypt, but it’s Cloudflare’s responsibility to contact their upstream provider about the routing problems. Nobody else can do it on behalf of Cloudflare. That’s just how it works.

Let’s assume the Egyptian Internet Authorities detected something malicious related to 188.114.97.6/31 a few months ago and rather than contact Cloudflare about the abusive website, they just lazily blocked both IPs.

The very act of Cloudflare proactively inquiring them about this problem will expose the root cause. If there truly was an excuse for blocking it, then Cloudflare can provide them with evidence that the abusive website has been resolved so there is no more reason for the blocking. Or if it was just false-positive blacklisted, then they can just repair it. Either way will allow routing to work properly once again.

This process could take days or weeks to figure out how to resolve. In the meantime, Cloudflare ought to resolve all DNS requests sourcing from Egypt to Djibouti or Kenya or Greece or literally ANY other data center besides Amsterdam until this known issue is resolved. The rest of the world can continue to DNS to Amsterdam.

And/or at least mark the AMS as “Partial Outage” until Cloudflare has time to fix the DNS to avoid this known issue. It’s affecting far too many people right now. Pretending like everything is “Operational” is causing even more frustration for everyone attempting to route through Egypt.

What do you have against Cloudflare trying to fix this anyways? I’m sure Cloudflare has a Network Team that deals with this kind of stuff every day. Right?

It’s not everyday that network operators don’t understand what impact blocking shared infrastructure has, no - have you made a support ticket to bring it to their attention?

It isn’t really a Cloudflare problem to ‘fix’ per-se - if an IP is blocked by local authorities, and you just rotate IPs to get around it, that could be taken the wrong way by the aforementioned local authorities leading to more blanket restrictions.

Curiously, where does AMS come into this? If you got that via GeoIP, it’s completely meaningless for an anycast network.

Every PoP advertises the same IPs - there is no geographic location for that single IP.

2 Likes

What makes you think Cloudflare is not?

Yes, let’s.

Well, no. There’s no reason to assume that a website assumed to be objectionable by the government of Egypt needs to be removed from the Internet. And if there is collateral damage from that, too bad for the citizens of Egypt? I guess you could take the problem up with the government.

Nah. They should acknowledge the censorship by certain governmental agencies to be sure and they are absolutely wrong not to. But if a government wants to use a sledgehammer to block content, ■■■■ em.

Oh, I just assumed it was AMS from here:

https://dnslytics.com/ip/188.114.97.6

And Cloudflare could use AnyCast for its DNS resolvers because that’s UDP. But HTTPS is TCP so that would be dangerous to try to AnyCast that, eh?

I assumed Cloudflare detects the source of who is doing the resolving and provides an IP based on that, because I get different “A” entries depending on WHERE in the world I do the lookup.

That’s the biggest problem with Cloudflare. It’s impossible to inform them there is an issue. I bet Cloudflare is completely oblivious to this problem.

But yes, I’ve already opened a ticket on the Egypt side. So they should be able to work there way towards each other from both ends until the broken transient ISP gets the complaints from both ends.

And there is not really any Cloudflare website that is actually being blocked by this Egyptian failure because there’s always multiple IPs and only the one IP is being filtered, so the client will eventually round robin to the other IP and still serve the content after choking for a while. This “sledgehammer” drop is obviously hurting the thousands of other websites that happen to also go through Cloudflare completely unrelated to whatever they think they are blocking. So if this filter WAS intentional, it’s broken due to not enough block and it’s broken due to too much block at the same time. Epic Failure. Whoever did this obviously doesn’t understand what they are doing. It just needs to be fixed.

Cloudflare’s obliviousness is a real problem honestly. If they don’t know this is an issue than that indicates poor monitoring and oversight. If they do know this is an issue, they should do something about it. At the very least, they should notify customers (especially though that get a high # of visitors from Egypt) that there are connection problems from Egypt (via email or a banner in the dashboard).

As things are, we’re stuck with confusing messages from our visitors, assuming that all is fine from Cloudflare’s side.

I don’t understand what UDP or TCP has to do with anycast.

Can’t say I’ve ever really experienced this - DNS Propagation Checker - Global DNS Testing Tool

And so if Cloudflare remove that IP from the pool and the website they wanted to block is available again then another IP will just get blocked. Cloudflare juggling IPs to avoid not-so-bright network operators playing whack-a-IP isn’t a solution.

If a large amount of visitors can’t reach the site, a large amount of complaints should be going to their ISPs.

2 Likes

Oh, that is a nifty tool. So if you try a Cloudflare enabled IP (like ipconfig dot io) then you’ll see it resolve to many different IPs based on where the query comes from.

This exactly proves my point that Cloudflare already has the capability to point to different IPs based on source location, yet one of the IPs that Cloudflare is pointing Egyptian traffic to is known to have routing problems with Egypt. Is Cloudflare intentionally complicit in this Egyptian routing problem? Or just oblivious?

Yes, this is true. But the main purpose to avoid the broken IP isn’t to play wack-o-mole. It’s just temporary while Cloudflare is waiting for the response from their transient provider. Remember that this can take time for the Egyptian router boys to investigate or figure out why the route is broken or wherever it really is. We MIGHT assume that someone was intending to block a website, but we don’t even know that. (Remember that only the IP is half route blocked but the websites are all still eventually working.) So even if the admins are idiots, they can see by reloading on their browser that the website still comes up even from Egypt, so that assumption theory really doesn’t make sense either. Thus it seems more likely to be an UN-intentional misconfiguration. We just don’t know where the problem is yet.

Is there anyone in the world that is strong enough to contact Cloudflare just to let them know? Is anyone at Cloudflare even aware that their tiny IP range (188.114.97.6 and 188.114.97.7) has problems from most Egypt routes? Has Cloudflare already figured out exactly WHERE the route is breaking? Has Cloudflare already attempted to contact their upstream provider?

Yes, I realize that we already have tons of piddly Egyptian data centers whining to their ISPs about Cloudflare being half blocked. But what we need is someone with a lot of network influence to start opening tickets and working on this problem from the other side. Isn’t Cloudflare big enough with their millions of websites?

True enough. If a Nation State decides to block an IP address is an encumbrance on the provider to figure out why? Too much ankle isn’t something that can be defined and if 10,000 legitimate sites are blocked in the process who are we to argue?

Exactly. If the inquiry comes from the actual owner of the IP that is blocked (188.114.97.6/31 = Cloudflare) and the inquiry actually reaches to the provider owning the root cause of the routing problems, then the communication will be 100% perfectly legitimized. Random complaints originating from anyone else carry very little sway. Until Cloudflare asks them, nobody will know if these routing malfunctions are intentional or just an accidental misconfiguration.

Can anyone please confirm Cloudflare is already aware of this Egypt problem? That’s the first step.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.