Cloudflare traffic routing in Australia

tl;dr we upgraded from the Pro to the Business Plan and are now seeing Australian traffic routed through Australian POPs (colo’s).

  • We noticed that Australian traffic to our api was very slow (> 700ms when routed via Cloudflare and ~70ms when bypassing Cloudflare)
  • Checking our api url with https://$API_URL/cdn-cgi/trace we noticed that much of our traffic was being routed via colo’s in Hong Kong (HKG) and Osaka (KIX) despite us being in Melbourne (Telstra was the ISP we were testing with)
  • Using a commercial VPN in Australia, we saw that our colo was now MEL and api request times went down to 70-80ms.
  • Based on the conversation here: Routing wrong country it was really unclear whether upgrading our plan would make any difference to where traffic gets routed or if it was simply and ISP issue.
  • After seeing this reply: Routing wrong country we decided to try the upgrade and we can 100% confirm that our traffic is now reliably getting routed via Australian POPs

I guess it’s disappointing that Cloudflare don’t say this anywhere (that I can find) on their website and in the above community thread say things like, “We no longer have the limitations I described previously (as I said at the time, things are/were in flux). At this point all colos (except China POPs) are are available on all plans. Specific connections are based on the peering decisions by individual network providers.” which doesn’t appear to be true.

Based on the detailed discussion in this post

2 Likes

What they say is true, your site CAN be served by the Melbourne PoP on the Free plan, but will it? Most likely not. As you found out by using a VPN, it also depends on the ISP your customers are connecting from. Optus and Telstra refuse to peer with Cloudflare, as detailed in https://blog.cloudflare.com/bandwidth-costs-around-the-world/

But I agree, this information could be definitely be made more clear

I’ve done some homework, and collated the results for some Aussie ISPs, connecting to digital.com/cdn-cgi/trace (a site known to be on the FREE plan), see table for info, COLO is the colo that they were connected to

ASN IP Address City Region COLO
AS10143 Exetel Pty Ltd 2406:3400:41e:300:50a4:c7ac:* Brisbane Queensland SYD
AS1221 Telstra Corporation Ltd 2001:8003:555d:de00:9968:52a6:* Melbourne Victoria MEL
AS1221 Telstra Corporation Ltd 2001:8003:7d48:1a00:9d53:299:* Brisbane Queensland SYD
AS1221 Telstra Corporation Ltd 120.155.* Brisbane Queensland KIX
AS1221 Telstra Corporation Ltd 120.153.* Canberra ACT KIX
AS1221 Telstra Corporation Ltd 101.186.* Melbourne Victoria KIX
AS1221 Telstra Corporation Ltd 101.177.* Brisbane Queensland KIX
AS1221 Telstra Corporation Ltd 2001:8003:6e0f:1a00:987:54df:* Brisbane Queensland SYD
AS1221 Telstra Corporation Ltd 60.226.* Brisbane Queensland KIX
AS1221 Telstra Corporation Ltd 2001:8003:c978:a000:d55e:3f76:* Charlestown New South Wales SYD
AS1221 Telstra Corporation Ltd 110.143.* Perth Western Australia KIX
AS133414 Foxtel Management Pty Ltd 172.197.* Melbourne Victoria KIX
AS133612 Vodafone Australia Pty Ltd 2405:6e00:2ff8:7001:8891:5cb2:* Melbourne Victoria MEL
AS133612 Vodafone Australia Pty Ltd 120.23.* Brisbane Queensland SYD
AS135887 Telstra Corporation 141.168.* Sydney New South Wales KIX
AS135887 Telstra Corporation 149.167.* Melbourne Victoria KIX
AS135887 Telstra Corporation 144.138.* Sydney New South Wales KIX
AS136972 Gigafy 103.100.* Brisbane Queensland SYD
AS137549 NODE1 Pty Ltd 103.100.* Perth Western Australia SYD
AS24093 SUPERLOOP (BIGAIR. Multihoming ASN) 125.253.* Brisbane Queensland SYD
AS4739 Internode Pty Ltd 150.101.* Perth Western Australia HKG
AS4739 Internode Pty Ltd 150.101.* Perth Western Australia HKG
AS4739 Internode Pty Ltd 121.45.* Launceston Tasmania MEL
AS4764 Aussie Broadband 180.150.* Melbourne Victoria MEL
AS4764 Aussie Broadband 167.179.* Morwell Victoria SYD
AS4764 Aussie Broadband 121.200.* Melbourne Victoria MEL
AS4804 Microplex PTY LTD 210.49.* Melbourne Victoria HKG
AS4804 Microplex PTY LTD 110.32.* Melbourne Victoria HKG
AS4804 Microplex PTY LTD 175.34.* Melbourne Victoria HKG
AS4804 Microplex PTY LTD 49.184.* Melbourne Victoria HKG
AS4804 Microplex PTY LTD 106.71.* Brisbane Queensland HKG
AS4804 Microplex PTY LTD 114.78.* Sydney New South Wales HKG
AS4826 Vocus Connect International Backbone 49.255.* Sydney New South Wales MEL
AS7545 TPG Telecom Limited 124.149.* Brisbane Queensland HKG
AS7545 TPG Telecom Limited 124.170.* Melbourne Victoria HKG
AS7545 TPG Telecom Limited 193.115.* Adelaide South Australia HKG
AS7545 TPG Telecom Limited 203.111.* Sydney New South Wales HKG
AS7545 TPG Telecom Limited 220.240.* Perth Western Australia HKG
AS7545 TPG Telecom Limited 14.203.* Sydney New South Wales HKG
AS7545 TPG Telecom Limited 118.211.* Adelaide South Australia HKG
AS7545 TPG Telecom Limited 59.102.* Melbourne Victoria HKG
AS7545 TPG Telecom Limited 110.174.* Sydney New South Wales HKG
AS7545 TPG Telecom Limited 118.209.* Sunshine Coast Queensland HKG
AS7575 Australian Academic and Research Network (AARNet) 134.148.* Newcastle New South Wales SYD
AS9310 MYREPUBLIC PTY LTD 202.169.* Perth Western Australia SYD
AS9310 MYREPUBLIC PTY LTD 103.217.* Sydney New South Wales SYD
2 Likes

Yeah I suppose I expect a lot from Cloudflare as I generally think they do a great job and provide a great free product. But this is essentially: if you have a website hosted in Australia and a target audience in Australia, Cloudflare will probably slow your website down considerably (not denying the security improvements, but performance is a significant part of their value proposition).

2 Likes

I had noticed the same thing on the free plan. Some ISPs in Australia (eg Telstra and Optus) traffic is routed to a Cloudflare PoP offshore even if the origin is in Australia, actually making performance worse - traffic potentially goes offshore and then back (large added RTT) even though there are PoPs in Australia.

You can bet that the ISP and Cloudflare will point the finger at each other for this type of routing being based on commercial reasons, noting that Telstra do appear to interconnect directly with Cloudflare, just not all the traffic is sent that way (for me traffic to/from 1.1.1.1 routes via a router called clo2241358.lnk.telstra.net [139.130.67.10] - clo almost certainly being Cloudflare).

However interestingly for Telstra, IPv6 is for me routed to the local AU PoP on the free tier. I’m surprised that a technical difference of an IP header would make a difference to commercials on interconnects but I have little knowledge of this area. Perhaps it’s about moderating the amount of traffic on these domestic interconnects.

@simon29, that’s a really interesting observation. We’re routing to a load balancer inside AWS so not quite sure how we can use ipv6 but will investigate it.

@misha2, IPv6 is used between the end-user accessing your site and the Cloudflare PoP, and for the free tier it’s on automatically and nothing to configure I don’t think it can be disabled. Perhaps on pro/business it can be turned on/off.

Even if your origin site supports only IPv4, IPv6 can still be used for the end user to the Cloudflare CDN.

I’m not sure how widespread IPv6 is in Australia, perhaps this may help, unsure how up to date it is.
https://whirlpool.net.au/wiki/hw_feature_242

Thanks for the clarification. I’m not entirely sure if this test is valid (a bit out of my wheel house) but I seem to get a remote POP even when using ipv6 (ISP Telstra in Melbourne, Aus):

$ curl $WEB/cdn-cgi/trace
fl=51f118
ip=60.xxx.xxx.xxx
ts=1592172071.059
visit_scheme=http
uag=curl/7.54.0
colo=KIX
http=http/1.1
loc=AU
tls=off
sni=off
warp=off
Mon Jun 15 08:01:11 [~]

$ curl -6 $WEB/cdn-cgi/trace
fl=51f71
ip=60.xxx.xxx.xxx
ts=1592172075.117
visit_scheme=http
uag=curl/7.54.0
colo=KIX
http=http/1.1
loc=AU
tls=off
sni=off
warp=off

And then connecting to VPN in Melbourne:

$ curl $WEB/cdn-cgi/trace
fl=47f43
ip=60.xxx.xxx.xxx
ts=1592172169.483
visit_scheme=http
uag=curl/7.54.0
colo=MEL
http=http/1.1
loc=AU
tls=off
sni=off
warp=off

@misha2 it doesn’t appear you’re using IPv6.

This is for me for a host on the free tier, ISP Telstra via NBN in Melbourne, AU also :slight_smile:, I’ve hidden parts of my IP. IPv6 routes to MEL, IPv4 routes to KIX (Japan). Origin is in Melbourne.

$ curl $WEB/cdn-cgi/trace
fl=47f37
ip=2001:8003:xxxx:xxxx:58f5:4819:7a0a:e86f
ts=1592174537.146
visit_scheme=http
uag=curl/7.58.0
colo=MEL
http=http/1.1
loc=AU
tls=off
sni=off
warp=off

$ curl -4 $WEB/cdn-cgi/trace
fl=51f93
ip=121.221.x.x
ts=1592174544.163
visit_scheme=http
uag=curl/7.58.0
colo=KIX
http=http/1.1
loc=AU
tls=off
sni=off
warp=off

and the Telstra 4G mobile network, also in Melbourne

$ curl $WEB/cdn-cgi/trace
fl=47f53
ip=2001:8004:xxxx:xxxx:f149:847b:2888:bfad
ts=1592173327.082
visit_scheme=http
uag=curl/7.58.0
colo=MEL
http=http/1.1
loc=AU
tls=off
sni=off
warp=off

$ curl -4 $WEB/cdn-cgi/trace
fl=51f95
ip=1.136.x.x
ts=1592173330.872
visit_scheme=http
uag=curl/7.58.0
colo=KIX
http=http/1.1
loc=AU
tls=off
sni=off
warp=off

You’re correct. The ipv4 address in the response should’ve been a hint :slight_smile:

I’m not able to get my Macbook Pro to use ipv6 (despite having an ipv6 address) but it’s also less interesting if it isn’t something we could do server side to get people onto ipv6 (and therefore pay less for Cloudflare).

This topic was automatically closed after 30 days. New replies are no longer allowed.