Routing in Russia

[anry]

Hello,

decided to check how Cloudflare CDN works in Russia. I have a site hosted in Saint-Petersburg location. According to the blog posts (Moscow, Russia: CloudFlare’s 83rd data center and Ten new data centers: Cloudflare expands global network to 165 cities) there is a data center in Moscow alongside POP in Saint-Petersburg, both of which have been created to avoid routing traffic from Russia to Stockholm and Frankfurt reducing such way a latency for the end users.

So far so good. I registered Cloudflare account and added my site into it. Everything is fine, works, etc. Special thanks for such a good TLS implementation with all fancy things.

But… I found that the site became more slow than it was before. I tried to ping it and found that latency increased more than twice (!) from 32 ms to 72 ms.

What I found later is even more interesting. I started to trace DCs (using /cdn-cgi/trace) which traffic is routed to from different parts of Russia. During the tests DME (Moscow) and LED (St.-Petersburg) DCs have status Operational according to Cloudflare System Status service. So,

Saint-Petersburg

fl=132f48
h=***.ru
ip=87.236.*.*
ts=1549787084.286
visit_scheme=https
uag=curl/7.22.0 
colo=KBP
http=http/1.1
loc=RU
tls=TLSv1.2
sni=plaintext

Moscow

fl=128f61 
h=***.ru 
ip=87.228.*.* 
ts=1549734931.65 
visit_scheme=https 
uag=Mozilla/5.0 (Linux; Android 8.0.0; PRA-TL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.99 Mobile Safari/537.36 
colo=ARN 
http=h2 
loc=RU 
tls=TLSv1.3 
sni=plaintext

Moscow (mobile operator's network)

fl=71f312 
h=***.ru 
ip=176.59.*.* 
ts=1549735170.907 
visit_scheme=https 
uag=Mozilla/5.0 (Linux; Android 8.0.0; PRA-TL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.99 Mobile Safari/537.36 
colo=FRA 
http=h2 
loc=RU 
tls=TLSv1.3 
sni=plaintext

Saratov

fl=71f158
h=***.ru
ip=79.126.*.*
ts=1549786975.214
visit_scheme=https
uag=curl/7.54.0
colo=FRA
http=h2
loc=RU
tls=TLSv1.2
sni=plaintext

According to these results, traffic from Saint-Petersburg has been routed to Kiev, Ukraine (KBP), from Moscow to Stockholm, Sweden (ARN) and Frankfurt, Germany (FRA), from Saratov to Frankfurt, Germany (FRA). None of the cases shown routing to Russia’s national DCs.

The same results I got using plain HTTP instead of HTTPS, keep trying for 2 (maybe 3) days.

Now I’d like to know whether it is normal (designed) routing behaviour or not. Latencies that increased at least twice after Cloudflare integration look disappointing.

Thanks!

[cbrandt]

This has to do with the ISPs and their peering policies. Some ISP peer with Cloudflare, others don’t, sending their users to another region (even another country) based on the IP address of your website.

In my case, visitors in my city in Brazil to my website (in Brazil) are always routed to MIA (Miami) when I use my home wi-fi (one ISP), but to GRU (São Paulo) when I use mobile (another ISP).

Also, there was a day when I noticed a slowdown in my site, and that day Clouflare Status showed 4 DCs in the U.S. being re-routed. Though none of them were MIA, my site on that day was being served out of SEA or IAD.

For more details on this issue:

Routing wrong country

Hi, My server is in Sydney Australia and Im in Perth Australia and ive notices over the last few weeks my sites load time has decreased dramatically. I couldn’t figure out what the issue was until i bypassed Cloudflare DNS and the site was loading significantly faster… Faster than Ive noticed for well over a year. I tested and found the following cloudlfare = 13.24 seconds no Cloudflare = 3.77 seconds almost 10 seconds faster without Cloudflare. It seems that there are edge servers in mult…

[anry]

@cbrandt, thanks for your interest to my post and help!

As far as I know, the host at Saint-Petersburg I send requests from, belongs to the network which has direct peering with Cloudflare since middle of 2016 (according to hosting provider announcement).

After my previous post I checked routing from the hosts I used previously, but for well-known site - medium.com. Medium also uses Cloudflare and I suppose they use some paid plan.

Surprisingly (or not), all my requests from Saint-Petersburg were routed to Saint-Petersburg’s colo (LED):

Trace

fl=158f6
h=medium.com
ip=87.236.*.*
ts=1549874617.386
visit_scheme=https
uag=curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.2j zlib/1.2.3.4 libidn/1.23 librtmp/2.3
colo=LED
http=http/1.1
loc=RU
tls=TLSv1.2
sni=plaintext

All my requests from Moscow’s ISP were routed to Moscow’s colo (DME):

Trace ISP1

fl=87f149
h=medium.com
ip=87.228.*.*
ts=1549828347.045
visit_scheme=https
uag=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
colo=DME
http=h2
loc=RU
tls=TLSv1.2
sni=plaintext

Trace ISP2

fl=87f193 
h=medium.com 
ip=176.59.*.* 
ts=1549829391.629 
visit_scheme=https 
uag=Mozilla/5.0 (Linux; Android 8.0.0; PRA-TL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.99 Mobile Safari/537.36 
colo=DME 
http=h2 
loc=RU 
tls=TLSv1.3 
sni=plaintext

From Saratov requests to Medium are still routed to Frankfurt, but sometimes to Paris.

Apparently, Cloudflare has different routing policies for different customers (at least in Russia). I don’t blame anyone, I understand it’s a business. But several months ago @cs-cf stated that

At this point all colos (except China POPs) are are available on all plans.

I can’t believe that it’s local ISPs decisions to route Medium’s traffic to nearest colos and route my site’s traffic to another countries.

@cs-cf, could you please give some details on this particular case? Does Cloudflare preserve DME and LED colos for business customers? If not, how is it possible that requests to different Cloudflare customers from the same hosts are routed to different colos?

Thanks!

[cbrandt]

@anry

As far as I know, the host at Saint-Petersburg I send requests from, belongs to the network which has direct peering with Cloudflare

But how about the ISP that you use to access the internet? And the ISP your visitors are most likely to use to access the internet?

[anry]

I believe it doesn’t affect my tests. Does it?

[cbrandt]

Please compare these two screen shots I just took using my iPhone. Same website, one connection via Wi-Fi (my landline telephone company), the other using another ISP over 4G. (The VPN only means I’m using 1.1.1.1, not a real VPN)

[cbrandt]

Please also see this:

Is the Sofia Edge server out of service?

Hi, I live in Sofia, Bulgaria and up until a week or two ago everything worked normally and I was served from the Cloudflare Sofia server. No every time I make a request I get served from the Prague Edge server. Why is that? Is the Sofia server still operational? Best regards, Lubo

[anry]

Yes, I understand your case - you are making two requests to the same site using different ISPs.

But my case is opposite - I make two request to different sites (both are Cloudflare-powered) using the same ISP, host, link, connection, etc. and then I see that requests are constantly routed to different DCs.