The negative impact of cloudflare's Israel POP

[bahat.gil]

Hi,

As I recently discovered (see attached RIPE ATLAS measurements), Cloudflare’s Israel POP serves less than half the country - namely, it’s designed to serve one monopolistic ISP and no one else, contributing to upsetting the delicate ISP balance here. This is dismaying and I hope Cloudflare would rectify this as soon as possible, both for the benefit of clients and end users alike.

In more detail:

Cloudflare apparently placed its presence in bezeq’s server farm. Bezeq is the largest telco in Israel and has been embroiled in monopolistic behaviour in the past. It’s competitors are struggling and it is opined that should bezeq be allowed to merge with its subsidiaries (ISP, Satellite TV provider, Cellular provider) it would be the death of competitive market for these services.

Bezeq has been called up on this particular behaviour in the past - when they did it with limelight networks. But limelight is a very small fry compared with Cloudflare. The recent Cloudflare outage has taken down a significant part of the internet.

So, in essence, this generally means Cloudflare has been contributing to the creation of a tiered, less open internet.

As an aside, this puts the implicit messaging as false advertising - the POP is more of a bezeq one than an Israeli one, even if it’s physically present at Israel, due to routing.

What also bothers me is that this situation may or may not happen in other markets experiencing similar conditions.
(as another aside, don’t your engineers break down reachability and routing by AS? surely someone should have seen that with more sophisticated tools than RIPE ATLAS. similar inquiries with other large companies have shown that they do have this data.)

I would be happy if you could advise soon. I’m available for any question or concern. I have considerable good faith in Cloudflare as a company in particular post its termination of the stormfront account and I hope it will continue to act in a similar, socially-responsible as well as business-prudent manner regarding this issue.

Gil

RIPE Atlas measurements - 1.1.1.1 and cdn.discordapp.com:

https://atlas.ripe.net/measurements/22303263/

https://atlas.ripe.net/measurements/22302685/

[domjh]

Hi @bahat.gil,

I see the situation here and understand your concern.

Your routing to Cloudflare’s servers is generally controlled by your ISP and their peering agreements.

You can also check out this community tutorial:

Peering - Why don't I reach the closest datacenter to me?

Why don’t I reach the closest datacenter to me? Cloudflare is an anycast network, so where you are routed is determined by peering with ISPs. The shortest path geographically isn’t always possible due to peering agreements. If you request a site hosted on Cloudflare from home: The domain resolves to a Cloudflare IP Your browser sends an HTTP request to that IP Your ISP’s routers consult their routing table and route the packets upstream to their transit provider ( COSTS MONEY ) The packets …

I see what you are saying, however I am unsure that Cloudflare will be able to do anything about it unless the other ISP’s get peering agreements.

Perhaps others here on the community will also have some insight into this.

[bahat.gil]

Hi,

no, this is cannot be rebuffed so simply. I would expect Cloudflare to intercede on its customer’s behalf and refuse to accept this behaviour by a local ISP. if I plan my network presence and see an Israel POP listed, I would naturally expect it to serve all of Israel and not half or less the country.

Cloudflare agreeing to such a horrendous deployment / peering agreement can have several implications, none of which are good looking for its customers and end users:

A. Cloudflare is unprofessional enough to not monitor the situation and realize that other AS’s aren’t being served by this POP. as bad as it may sound, this is the worse of evils.

B. Cloudflare has willingly chosen to ignore the massive negative impact its decision has on neighbouring ASes, the local fledging IXP and the local ISP ecosystem. If that is the case, every Israeli citizen should hold the company accountable on this - this notwithstanding if it goes kowtowing to such mannerisms in other markets as well. For a company that wants to “do no harm” to the global internet, this behaviour looks like massive harm to me - and Cloudflare’s only motivation to do this would be to save a few bucks.

not good at all.

[domjh]

I never said that I agree with it or that I think they made the right decision here. I don’t speak for Cloudflare.

I was just pointing out how it works - it is the same in many cases, not a unique issue!

@bahat.gil

expect it to serve all of Israel

This is not the case in may places, it often depends on ISP where you are routed and it is quite common to be routed outside your own country.

Might I suggest you contact the other ISPs and ask them about their peering agreements?

Partners Peering Portal

I doubt Cloudflare can control whether the other ISPs choose to set up peering agreements with them.

Again, just to reiterate - I am not saying that Cloudflare plays no part in this, nor an I saying they do. I am just explaining why this happens, from my point of view.

[bahat.gil]

I see. I would wager that you are not from a country which is an ‘internet island’ the way Israel is.

As to your question about peering: All local ISPs are peered with each other - either via direct peering or a local IXP called IIX - Israel Internet Exchange. It’s even mandated by law, iirc.

I find it extremely hard to believe that Cloudflare has little or no say with how its presence is further routed towards neighbouring ASes. In which case, Cloudflare could have selected another hosting site that would serve thrice the user base for the same POP.

I have plenty of goodwill towards Cloudflare, which is why I’m eager to get an official response on this.

[boynet2]

a few things:

1. Cloudflare is fully aware of it, you can see Cloudflare employee saying it in the blog comments Tel Aviv, Israel: Cloudflare's 135th Data Center Now Live!

Which ISP are you using? Our partnership currently serves Bezeq (and their customers). We expect to interconnect with additional networks through the year.

If you’re on Bezeq and having trouble, email us a traceroute to support@, and we’ll take a look. Thanks!

2. from my data, around 35% of Israeli users are hitting Israeli pop, the rest hitting france london and amsterdam

3. you are false assuming that its only Israel problem, I believe its happening in multiple pops.

4. but on the bright side even if you hit France, 60ms is not that bad or something that most people will even notice, and you know how pricey is the bandwidth in Israel and 35% of users will enjoy really fast connection is better than nothing

[bahat.gil]

1. Okay, so that rules out option #1. it’s been a year and there have been no interconnects. Again, we’re not talking about erecting any more physical lines needed, this was something that ostensibly should have taken a few weeks at most.
   everyone would rather peer with Cloudflare given the option, existing underutilized lines (IIX) and being a prominent content network. I am at a loss to find a plausible explanation for this that is innocuous.

2. Yep, that makes sense. Have you considered the impact on the local ISP market? Net neutrality as a moral consideration?

3. Re: 2 - all the more so? If content networks aren’t going to stand up for net neutrality and the benefit of consumers, who would?

4. No. I would rather have net neutrality and a functional local market than allow a single AS to leverage its monopolistic power. (and a connection to the IIX is not that pricey, mind you). I would expect a company to understand this.

[cloonan]

@bahat.gil

official response

Hi @bahat.gil, the Community of Cloudflare users cannot offer an official statement. Your comments will get more visibility and are better placed on the post @boynet2 shared.