Problem setting up load balancers, no visibility on cloudflare side

[pierre3]

I am trying to setup a load balancer correctly but I can hardly get any feedback from what is happening on Cloudflare Load Balancer.

All Pools and Origins are showen healthy.

I choose Dynamic Steering, but I do not know the result of Cloudflare pinging those servers and the preferred servers that Cloudflare choose for each region.

I currently have 3 pools:
pool_us-east-1 => origin_us-east-1
pool_us-east-2 => origin_us-east-2
pool_ap-southeast-1 => origin_ap-southeast-1

I do not get any traffic to us-east-2 (looking at my servers log because I do not have any feedback from Cloudflare about what is happening)

I tried this:
pool_us-east-1 => 1 origin_us-east-1 weight:0.5 AND 1 origin_us-east-2 weight:0.5
pool_us-east-2 => origin_us-east-2
pool_ap-southeast-1 => origin_ap-southeast-1

I still get no traffic to us-east-2

The weirdest part is that if I disable pool_us-east-1 (off) I still get the traffic to us-east-1…

Can anyone help with my issue?
I am wondering if there are any plans from Cloudflare to offer more visibility about what is happening with the load balancer traffic?

[eva2000]

Been a while since i played with Cloudflare Load balancer. But have you properly setup cf health checks to fail over https://support.cloudflare.com/hc/en-us/articles/115000081911-Configuring-Load-Balancing-Intelligent-Failover ? What criteria are you using for cf pool’s health checks to determine failover triggers ?

How are you testing and determining where visitor requests are being routed to ? Dynamic steering routes requests based on measure latency to the fastest cf load balancer pool as opposed to Geo steering which routes based on Cloudflare region serving the request. So with Dynamic steering, if origin_us-east-1 is faster than origin_us-east-2 for specific visitors, then cf load balancer will always route to origin_us-east-1.

As Cloudflare Load balancer configuration can be highly technical and we don’t have access to your configuration, it’s best to contact Cloudflare tech support to ask

[pierre3]

Thank you for your answer.
I am looking at my servers log of my 3 servers to see which one get calls.

My Health checks are all green. They basically test a GET endpoint expecting 200 and a response containing the string value data.
I am not using pool as fail over it is more about directing traffic to the fastest endpoint.

You are right that is what I thought, us-east-1 is faster than us-east-2.
I decided to add us-east-1 as a 0.5 weighted origin of pool_us-east-1. In this case the traffic should be spread through those 2 origins without dynamic steering (I think).

I am frustrated that Cloudflare does not provide more insight about how traffic get redirected and about the health checks speed seen by Cloudflare. Cloudflare could display the result of querying those monitor from each region.
Moreover, the logs reaching the server does not tell me the user location and the route took by the load-balancer.
Cloudflare could offer a way to visualise the load balanced users. Even something as simple as filtering by a timestamp and seeing the percent of the traffic received by each pool would be helpful.

Maybe it is possible to get those information but I am just not aware on how to get them.
Thanks

[eva2000]

@pierre3

Moreover, the logs reaching the server does not tell me the user location and the route took by the load-balancer.

You can try custom logging at origin server end of cloudflare’s IATA datacenter codes. This is what I do on my custom nginx origin side outlined at https://community.centminmod.com/threads/cloudflare-custom-nginx-logging.14790/

example from my nginx origin logs

tail -1 /home/nginx/domains/domain.com/log/cfssl-access.log | awk '{print NR": "$0; for(i=1;i<=NF;++i)print ""i":  "$i}'
1: 180.76.15.158 - - [13/May/2018:22:41:32 +0000] GET /tags/php-54/ HTTP/1.1 "200" 45399 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "180.76.15.158" "-" "39" "1" "0.100" 41a8a20c1c953343-HKG TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
1:  180.76.15.158
2:  -
3:  -
4:  [13/May/2018:22:41:32
5:  +0000]
6:  GET
7:  /tags/php-54/
8:  HTTP/1.1
9:  "200"
10:  45399
11:  "-"
12:  "Mozilla/5.0
13:  (compatible;
14:  Baiduspider/2.0;
15:  +http://www.baidu.com/search/spider.html)"
16:  "180.76.15.158"
17:  "-"
18:  "39"
19:  "1"
20:  "0.100"
21:  41a8a20c1c953343-HKG
22:  TLSv1.2
23:  ECDHE-RSA-AES256-GCM-SHA384

summary of cloudflare IATA datacenter codes that served my visitors

pzcat -f /home/nginx/domains/domain.com/log/cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn | awk '{print $1, $2, "https://google.com/search?q=iata+code:+"$2}'    

398 YVR https://google.com/search?q=iata+code:+YVR
390 DME https://google.com/search?q=iata+code:+DME
389 NRT https://google.com/search?q=iata+code:+NRT
209 SIN https://google.com/search?q=iata+code:+SIN
171 DFW https://google.com/search?q=iata+code:+DFW
152  https://google.com/search?q=iata+code:+
146 AMS https://google.com/search?q=iata+code:+AMS
134 IAD https://google.com/search?q=iata+code:+IAD
130 SJC https://google.com/search?q=iata+code:+SJC
116 FRA https://google.com/search?q=iata+code:+FRA
114 LHR https://google.com/search?q=iata+code:+LHR
99 ORD https://google.com/search?q=iata+code:+ORD
99 EWR https://google.com/search?q=iata+code:+EWR
72 PDX https://google.com/search?q=iata+code:+PDX
53 GRU https://google.com/search?q=iata+code:+GRU
41 HKG https://google.com/search?q=iata+code:+HKG
41 BOM https://google.com/search?q=iata+code:+BOM
29 LAX https://google.com/search?q=iata+code:+LAX
28 MEL https://google.com/search?q=iata+code:+MEL

summary of SSL protocol/ciphers used to serve visitors

pzcat -f cfssl-access.log{,-*} | grep -v 'SERVER_IP' | awk '{n = 2; for (--n; n >= 0; n--){ printf "%s\t",$(NF-n)} print ""}' | sort | uniq -c | sort -rn | head -n20 | column -t
299930  TLSv1.2  ECDHE-ECDSA-AES256-GCM-SHA384
191033  TLSv1.2  ECDHE-ECDSA-CHACHA20-POLY1305
173484  TLSv1.3  TLS_AES_256_GCM_SHA384
43      -        -
35      TLSv1.2  ECDHE-ECDSA-AES128-GCM-SHA256

You can also on your origin servers add a custom HTTP header tagging which server is used to serve the request.

Also check if Cloudflare zone analytics API has any info you can use Cloudflare API Documentation - I don’t use Cloudflare Load Balancer right now so can’t check myself. I only used it when it was early beta testing when it was non-gui/API only ~3yrs ago.

Maybe someone else here with current Load balancer experience can provide more insight. But definitely as cloudflare tech support too as they will know best.

[pierre3]

Thanks for the pointer to Cloudflare zone analytics API I will try that.

I had an issue in my setup that caused traffic to keep going to us-east-1 but I fixed it.
I can now work on other things but I still thing that we should have more analytics about what is happening with the load balancer.
If anyone has a good curl command to query the zone analytics API for answering question like:

• how much traffic got directed to pool1? to pool2?
• how many users from the US got directed to pool1? to pool2?
• what is the current response time from Cloudflare Northeast US to query my pool1 monitor? pool2 monitor?

Thanks