Seeing latency on CF nodes when proxying traffic through CF

Hello,

I am proxying the DNS for one of my domains through CF. And I am seeing a lot of latency when communicating with that server.

My server finishes the work in 90ms but the total time taken on the browser shows as ~ 900ms. I checked with our cloud provider (DO) and they mention that there is no problem at their end.

Here is the output of running an MTR to our domain:

shrayasr@shrayasr-hp:~$ mtr -rwbzc 100 lspl.fairmpos.com
Start: 2024-05-28T10:37:10+0530
HOST: shrayasr-hp                                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    shrayasr-hp.mshome.net (192.168.192.1)                             0.0%   100    1.0   0.8   0.2   4.3   0.5
  2. AS???    192.168.68.1                                                       0.0%   100    3.5   3.8   2.1   8.0   0.9
  3. AS???    local.airtelfiber.com (192.168.1.1)                                0.0%   100    4.7   4.8   3.2  10.6   1.2
  4. AS24560  abts-tn-dynamic-001.48.164.122.airtelbroadband.in (122.164.48.1)   0.0%   100   13.0   8.6   4.7  47.0   6.3
  5. AS9498   125.18.37.157                                                     40.0%   100    8.4  15.3   4.8 124.4  21.9
  6. AS???    182.79.198.129                                                     0.0%   100  137.5 138.4 134.3 164.9   4.6
  7. AS13335  162.158.20.18                                                     67.0%   100  139.0 144.0 135.1 173.2  10.1
  8. AS13335  162.158.20.240                                                     0.0%   100  136.2 142.7 134.5 205.3  12.5
  9. AS13335  172.67.190.185                                                     0.0%   100  139.8 137.0 134.5 150.9   1.9

All the hops with AS13335 seem to be taking a lot of time as far as I can understand. Is there someone that can help me understand (and maybe debug this) more?

If you’re looking at #6, that is a Bharti Airtel IP address as well, and that one is the one that is “starting” the high latency.

Bharti Airtel is taking you all the way, apparently from India, and to a Cloudflare PoP in Marseille, France.

There are some incumbents out there, that do not want to establish settlement-free peering (e.g. direct connections) with other networks locally (e.g. within their home markets).

As Bharti Airtel do not appear to want to establish a such settlement-free peering with Cloudflare locally (e.g. within India), Bharti Airtel is instead taking the traffic all the way to Europe, to pass it on to Cloudflare there.

It is a bad decision from Bharti Airtel, as well as other companies doing similar decisions, as it is ending up on hurting their own customer’s (e.g. your) connectivity to other networks.

1 Like

Bharti Airtel is taking you all the way, apparently from India , and to a Cloudflare PoP in Marseille, France .

Wow. But why would it do that when my datacenter of the terminal node is Bangalore, India? Literally 400 kilometers from where I am sitting right now.

It is a bad decision from Bharti Airtel, as well as other companies doing similar decisions, as it is ending up on hurting their own customer’s (e.g. your) connectivity to other networks.

This is really upsetting. To understand better, this means that if I proxy my DNS through CF for my webapp, and the client that’s consuming my service is on Airtel’s connection lines (or happen to hop through it), that they will face the same level of latency that I am currently facing?

And that too because Airtel refuses to get into some kind of a contract with CF for this?

It is a longer story, you can distribute traffic based in multiple ways - and every single way does, to some degree come with both pro’s and con’s.

Cloudflare broadcast their sets of IP addresses (the same IP addresses) from multiple locations worldwide, using a technique commonly referred to as “Anycast”.

Anycast” is good for distributing the the traffic towards multiple nodes (e.g. different datacenters), and ensuring that if one datacenter has to be taken offline, then because you run the same IP addresses “everywhere”, the traffic will just flow to another one.

If each individual server has a 10 Gbps connection, and you have 10 locations, it will literally be a total capacity of 100 Gbps (assuming that traffic is evenly distributed, which it unfortunately isn’t always), taking offline one of the 10 Gbps servers, you would obviously reduce capacity, but it wouldn’t take the IP address(es) completely offline.

One of the caveats that come with using “Anycast”, will be that it will be the source network, e.g. Bharti Airtel in this case, and their routing policy (regardless of whether it will be good or bad one), that will define where you end up.

An alternative way for traffic distribution would often be referred to as “GeoDNS”, which is a way where you make the authoritative name servers (DNS) of your domain have a IP address ↔ location database, which it uses to give different responses to the DNS queries, depending on where it believes the source of the traffic is from.

If the source appears to be India, it could for example respond with 203.0.113.147.

However, if the source appears to be in (or near Marseille, France), the authoritative name server could respond with 192.0.2.14.

This way would, to some degree, offer much more granularity, but it similarly comes with a caveat, - DNS cache being one of them.

The authoritative name server is responding with a low TTL (Time To Live), of 300 seconds (5 minutes), to allow for quicker migrations, if necessary, however, to avoid having to be boarding third parties with DNS querieas, your ISP has raised the minimum TTL (Time To Live) in their DNS resolver to 3600 seconds (1 hour).

There is currently an issue on the network where 203.0.113.147, but due to the DNS cache of your ISP (and their bad decisions in the above example), taking out 203.0.113.147 of the network would mean downtime for you, for at least up to the TTL (Time To Live) of 3600 seconds (1 hour).

In addition, distributing via “GeoDNS” does not extend the capacity for the same IP addresses as “Anycast” does. So “GeoDNS” may not be as resilient against e.g. DDoS attacks, as “Anycast” would.

Note: None of these two mentioned ways are 100% perfect, and both of them unfortunately come with both pro’s and con’s.

I completely agree.

That is true, -

Without Cloudflare, or with Unproxied (:grey:) / DNS-only records: Visitor ↔ Web server
With Proxied (:orange:) records: Visitor ↔ Cloudflare ↔ Web server

So, by having Proxied (:orange:) records, you also have two different connections, that are both depending on somewhat good connectivity, to have the best possible user experience.

Contracts are generally not necessary for establishing settlement-free peering, however, there are some ISP’s that may require one. I doubt Cloudflare requires such.

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.