Cloudflare Latency increments

Hi, We are running some latency tests on cloudflare pro as we are looking into using cloudflare for security purposes. As a prerequisite we have migrated all our DNS to cloudflare which is working fine. I am in India and so is our infra. Our customers however can be from anywhere

I ran the the tests on free plan at the beginning and it reported latencies of 600ms for HTTP and upwards of one second for TLS. I observed that requests were routing outside of India. Then we moved to pro plan and I saw that requests were now being routed in India but latencies ranged from 180 to 280ms for non TLS and 200 to 355ms for TLS connections. Depending on the time of the day it reports different latencies. Below are few pointers that mentions our testing setup:

  • We removed our app from the picture and created a static endpoint on our ingress to discount app processing.
  • We removed the default caching that Cloudflare does
  • Curl and Apache benchmark were used interchangeably to test out endpoints multiple times to derive proper numbers along with percentile latencies.
  • for both curl and AB, we are not using keep alive so no connection pooling

I have below questions:

  1. Why this latency goes in range throughout the day if requests are routed from India? What’s the request routing pattern that cloudflare follows?
  2. What we can to improve these latencies? Is there any setup change that we need?

Can anyone please help here? I see no response and we are blocked. We moved to business plan as well but it has not really helped us improve latencies. Latencies are similar to Pro plan

Look at the route from your connection to Cloudflare (check which data centre is in use by using https://www.cloudflare.com/cdn-cgi/trace) to see if “within India” is a near or far city. Also look at the route from your origin to Cloudflare.

If the route is long or has many hops, likely the peering policy of your ISP, or your origin’s ISP, or both is to blame and uses long or slow routes.

Thanks sjr for the response.

Trace call shows that requests from my home are within India and so is our infra. Within India I observe that location keeps changing during the day. Some times it is near to origin and sometime it is far. But since we are now on business plan my assumption was that requests will be routed on priority. Without cloudflare proxying it works fine.

Our app is used from different places in India as well as globally. My assumption was moving to higher plan will provide us tolerable latencies but that is not the case. Since end user’s ISP or origin’s ISP(Cloud provider) is not in our control. How can we make sure cloudflare latencies are within range? Also caching is not an option for us since majority of traffic is API.

Cloudflare cannot control the peering policy of the ISPs. Cloudflare offers free and open peering to anyone that wants to, but you can’t force traffic to take a route your ISP doesn’t provide. It’s a known problem in India, such as here…

https://broadband.forum/threads/can-somebody-contact-airtel-to-fix-their-cloudflare-peer-issue.221867/

Sure. I agree with that. But I checked in with my team mates in different cities and they are reporting numbers similar to me. Also, team mates in Africa are reporting worst numbers like latencies upto 1 Sec even on business plan. In addition to this, here is the output of traceroute from my machine for one of our proxied domains:

reliance.reliance (192.168.29.1) 2.654 ms 2.433 ms 2.057 ms
2 10.3.112.1 (10.3.112.1) 4.319 ms 3.647 ms 3.773 ms
3 172.16.25.5 (172.16.25.5) 7.167 ms 7.420 ms 6.624 ms
4 192.168.74.190 (192.168.74.190) 6.589 ms
192.168.74.196 (192.168.74.196) 7.240 ms
192.168.74.190 (192.168.74.190) 7.275 ms
5 172.26.100.228 (172.26.100.228) 7.801 ms 6.497 ms 6.764 ms
6 172.26.100.211 (172.26.100.211) 7.153 ms 8.209 ms 8.037 ms
7 192.168.38.23 (192.168.38.23) 8.143 ms
192.168.38.29 (192.168.38.29) 7.435 ms
192.168.38.25 (192.168.38.25) 8.325 ms
8 192.168.38.28 (192.168.38.28) 8.457 ms 10.774 ms
192.168.38.24 (192.168.38.24) 7.713 ms
9 172.16.25.4 (172.16.25.4) 16.149 ms
172.26.40.5 (172.26.40.5) 16.798 ms
172.16.92.147 (172.16.92.147) 17.330 ms
10 172.16.1.220 (172.16.1.220) 17.369 ms
172.26.40.5 (172.26.40.5) 17.017 ms
172.16.92.145 (172.16.92.145) 18.826 ms
11 49.44.187.43 (49.44.187.43) 17.087 ms
172.26.40.7 (172.26.40.7) 19.823 ms
172.16.92.145 (172.16.92.145) 17.885 ms
12 104.22.40.86 (104.22.40.86) 17.768 ms 17.437 ms 16.681 ms

It does not seem to be taking too much time. So my guess is slow down happens on of these stages:

  1. When CF tries to decrypt my TLS request
  2. When CF routes it to my origin and waits for the response. I am guessing routing from CF DCs will happen based on their ISPs and not mine.

My Questions are these:

  1. My assumption was moving to higher plan CF will not only route requests local to client’s country but will also provide it priority in processing and routing it from DC to origin. That does not seem to be happening at least not when I jumped from pro to business. Is this correct assumption?

  2. Given ISPs of end customers, CF and my origin are not in control and also caching is not an option is there any setup change I can do on CF side that will help me bring this lower? We like CF but this is really blocking us from migrating over.

It will be based on the ISP for the origin server. There are two sides to the connection:
client ↔ Cloudflare and Cloudflare ↔ origin. Either or both of these may have poor routing, congested links or others.

Cloudflare doesn’t have an ISP, it is one. Equipment goes into data centres and people can peer directly, or through peering points for free. If they don’t, or charge high fees to do so, there’s nothing Cloudlfare can do, short of you using Argo (which I don’t think will improve things within India) to make the traffic take a more direct route.

In London, I am 10ms from Cloudflare through my broadband ISP and my business origins are 0.8ms from Cloudflare (same, or next door data centre) and personal ones 2ms away due to direct peering between the ISPs and Cloudflare.

Thanks I was planning to give Argo a try. I see that lot of companies in india are using CF including the ones who are API first companies. What would be a general set of changes they must be doing to make it work if not Argo?