Cloudflare makes APIs super slow - even with Argo enabled

Our website is https://www.addy.co.nz/, providing address autocomplete and address validation to New Zealand customers. The predictive, suggest-as-you-type address autocomplete API should be really fast to provide a great user experience.

We’ve started using Cloudflare for DNS and load balancing. We’ve also enabled Argo for improved performance.

1. No Cloud Workers - Argo Enabled - Orange Cloud Enabled

It takes between 300 to 1100ms to complete an address search request when traffic is routed via Cloudflare.

2. Cloud Workers for Caching - Argo Enabled - Orange Cloud Enabled

It takes between 800ms to 1300ms to complete an address search request when hitting the Cloudflare cache to fetch requests.

3. No Cloudflare - Direct Origin Access

Bypassing Cloudflare and making requests to origin servers in Melbourne and Sydney directly takes between 45 to 70ms to complete from New Zealand on a fiber connection.

Observation

It appears like Cloudflare adds a huge latency performance tax of at least 300ms+ per request.

Is this type of performance expected? (e.g. 500ms instead of 50ms?)
Did we configure something wrong, which could cause the poor performance?
Are other experiencing the same performance?
What is the expected before and after Cloudflare performance?

1 Like

I take it you are referring to the request to https://api.addy.co.nz/search, right?

You’ll hardly be able to get 50 milliseconds, as you go through a proxy tunnel after all and that simply does add additional hops and work. It shouldnt take a second however. Can you post the output of https://api.addy.co.nz/cdn-cgi/trace?

One weird thing is that I, too, experienced quite a delayed response time (~1200ms), when it suddenly jumped down to 300ms. Have a look at these timings and ignore the favicon requests.

Could it be you have more than one record configured on Cloudflare for api?

Hi Sandro,

Thanks for looking into it. You are right, it is the /search API.

Here is the response from Fiddler. Using api.addy.co.nz/search (Cloudflare load balanced) is a lot slower compared to hitting the Melbourne or Sydney servers from New Zealand.

Here is the output from the cgi path:
fl=12f150
h=api.addy.co.nz
ip=121.74.99.195
ts=1560668088.105
visit_scheme=https
uag=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36
colo=LAX
http=http/2
loc=NZ
tls=TLSv1.3
sni=plaintext
warp=off

We are using the Cloudflare load balancer to route traffic to the origin servers (image 2).

I’ve even turned off all the caching and everything else, still terrible performance (image 3).

How much latency does Cloudflare, used as a reverse proxy, typically add to a request? I wouldn’t have thought it would make such a big difference.

Cheers,
Jay

no one can answer that as it different for each user, it depends on the user location, the user internet provider, the Cloudflare settings, the origin server location etc…

in your case looks like you are routed through Los Angeles, I have no idea what is the ping between New Zealand and Los Angeles but it doesn’t sound optimal

but I don’t think you can do anything about it, as you have no control about how your traffic is routed(but its always good idea to contact the support).

if its important to you maybe the best option for you is to add another server in usa

You appear to be routed via the US, Los Angeles in particular. That could easily explain the added latency. However it would not explain the difference in response times.

Your “api” host seems to go to a load balancer. I would not be surprised if the machines behind might show a different behaviour in terms of response times.

It probably comes down to the roundtrip which is happening in your case. Instead of going straight to the server, you are first being routed to the US, where the proxy then forwards the request back to New Zealand, only to have the response go back via the same route.

To confirm that, could you post the output of ping api.addy.co.nz?

Thanks for looking into it.

Addy is a New Zealand service for New Zealanders. There is no point adding a server in the US to serve NZ customers - hence the servers being hosted in the closest region, which is Australia.

I wouldn’t expect an exact answer on the latency, I was after an estimate. Surely people have looked into before and after response times.

In our case, Cloudflare sounds like a terrible solution (when going via LAX). Not sure how Argo is working when there is a Cloudflare edge node in NZ and traffic is routed via the US.

but if you do want to keep using Cloudflare, it sound for me like the best solution… It could reduce in half the latency.

I would talk with the support maybe they could route your site to better datacenter or something

Is there a way to prevent the route to go via the US?

Pinging api.addy.co.nz [104.24.108.7] with 32 bytes of data:
Reply from 104.24.108.7: bytes=32 time=127ms TTL=60
Reply from 104.24.108.7: bytes=32 time=128ms TTL=60
Reply from 104.24.108.7: bytes=32 time=127ms TTL=60
Reply from 104.24.108.7: bytes=32 time=131ms TTL=60

And tracert

3 5 ms 5 ms 2 ms 10.200.12.117
4 129 ms 130 ms 130 ms 38.88.197.206
5 128 ms 130 ms 130 ms te0-1-0-1.ccr41.lax04.atlas.cogentco.com [38.88.197.205]
6 128 ms 130 ms 131 ms be3271.ccr41.lax01.atlas.cogentco.com [154.54.42.101]
7 130 ms 129 ms 130 ms be3243.ccr41.lax05.atlas.cogentco.com [154.54.27.118]
8 130 ms 131 ms 130 ms 38.104.84.254
9 130 ms 130 ms 129 ms 104.24.108.7

tracert to the origin server:
3 2 ms 2 ms 2 ms 10.200.12.73
4 5 ms 6 ms 2 ms vodafonenz.akl01-96cbe-1a.ntwk.msn.net [207.46.36.198]
5 5 ms 6 ms 6 ms ae9-0.akl01-96cbe-1a.ntwk.msn.net [207.46.36.197]
6 5 ms 6 ms 6 ms ae21-0.ier02.akl30.ntwk.msn.net [104.44.233.225]
7 28 ms 30 ms 30 ms 104.44.233.214
8 41 ms 42 ms 42 ms be-102-0.ibr01.syd03.ntwk.msn.net [104.44.11.89]
9 43 ms 42 ms 42 ms be-6-0.ibr01.mel01.ntwk.msn.net [104.44.10.34]
10 44 ms 44 ms 42 ms ae101-0.icr01.mel20.ntwk.msn.net [104.44.11.158]

I’ve just tried it via another device, going through BNE (Brisbane I presume) and with that I’m getting 100 to 150ms response time. Much much better than the 300 to 800ms via LAX.

150ms is pretty responsive, not as good as 50ms, although something I think users can live with.

Does anyone know how Cloudflare decides which route a user should take? Going via LAX is far compared to the NZ or AU edge servers.

I believe I found the answer here:

It is a peering issue and only your ISP can fix that.

This topic was automatically closed after 30 days. New replies are no longer allowed.