Our website is https://www.addy.co.nz/, providing address autocomplete and address validation to New Zealand customers. The predictive, suggest-as-you-type address autocomplete API should be really fast to provide a great user experience.
We’ve started using Cloudflare for DNS and load balancing. We’ve also enabled Argo for improved performance.
It takes between 800ms to 1300ms to complete an address search request when hitting the Cloudflare cache to fetch requests.
3. No Cloudflare - Direct Origin Access
Bypassing Cloudflare and making requests to origin servers in Melbourne and Sydney directly takes between 45 to 70ms to complete from New Zealand on a fiber connection.
Observation
It appears like Cloudflare adds a huge latency performance tax of at least 300ms+ per request.
Is this type of performance expected? (e.g. 500ms instead of 50ms?)
Did we configure something wrong, which could cause the poor performance?
Are other experiencing the same performance?
What is the expected before and after Cloudflare performance?
I take it you are referring to the request to https://api.addy.co.nz/search, right?
You’ll hardly be able to get 50 milliseconds, as you go through a proxy tunnel after all and that simply does add additional hops and work. It shouldnt take a second however. Can you post the output of https://api.addy.co.nz/cdn-cgi/trace?
One weird thing is that I, too, experienced quite a delayed response time (~1200ms), when it suddenly jumped down to 300ms. Have a look at these timings and ignore the favicon requests.
Thanks for looking into it. You are right, it is the /search API.
Here is the response from Fiddler. Using api.addy.co.nz/search (Cloudflare load balanced) is a lot slower compared to hitting the Melbourne or Sydney servers from New Zealand.
Here is the output from the cgi path:
fl=12f150
h=api.addy.co.nz
ip=121.74.99.195
ts=1560668088.105
visit_scheme=https
uag=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36
colo=LAX
http=http/2
loc=NZ
tls=TLSv1.3
sni=plaintext
warp=off
We are using the Cloudflare load balancer to route traffic to the origin servers (image 2).
I’ve even turned off all the caching and everything else, still terrible performance (image 3).
How much latency does Cloudflare, used as a reverse proxy, typically add to a request? I wouldn’t have thought it would make such a big difference.
no one can answer that as it different for each user, it depends on the user location, the user internet provider, the Cloudflare settings, the origin server location etc…
in your case looks like you are routed through Los Angeles, I have no idea what is the ping between New Zealand and Los Angeles but it doesn’t sound optimal
but I don’t think you can do anything about it, as you have no control about how your traffic is routed(but its always good idea to contact the support).
if its important to you maybe the best option for you is to add another server in usa
You appear to be routed via the US, Los Angeles in particular. That could easily explain the added latency. However it would not explain the difference in response times.
Your “api” host seems to go to a load balancer. I would not be surprised if the machines behind might show a different behaviour in terms of response times.
It probably comes down to the roundtrip which is happening in your case. Instead of going straight to the server, you are first being routed to the US, where the proxy then forwards the request back to New Zealand, only to have the response go back via the same route.
To confirm that, could you post the output of ping api.addy.co.nz?
Addy is a New Zealand service for New Zealanders. There is no point adding a server in the US to serve NZ customers - hence the servers being hosted in the closest region, which is Australia.
I wouldn’t expect an exact answer on the latency, I was after an estimate. Surely people have looked into before and after response times.
In our case, Cloudflare sounds like a terrible solution (when going via LAX). Not sure how Argo is working when there is a Cloudflare edge node in NZ and traffic is routed via the US.
I’ve just tried it via another device, going through BNE (Brisbane I presume) and with that I’m getting 100 to 150ms response time. Much much better than the 300 to 800ms via LAX.
150ms is pretty responsive, not as good as 50ms, although something I think users can live with.
Does anyone know how Cloudflare decides which route a user should take? Going via LAX is far compared to the NZ or AU edge servers.
