Being very slow with AWS ELB when using CF load balancer

Hi,

We use CF load balancer to distribute traffic between Heroku and AWS.

Here is the thing:

  1. Request distributed to AWS was very slow, it usually took like 8s, 16s, 32s… But according to the logs of AWS, there is no slow requests at all. So, I guess the time cost was between CF LB and AWS LB, maybe creating the connections. Also, the request time is weird, like some kind of retry.
  2. When I set Heroku origin’s weight to 0, I got the same result.
  3. I tried to allow all :443,:80 traffic to AWS LB, got the same result.
  4. When I turn off the CF load balancer and resolve the DNS to AWS LB DNS name, all request were very fast.

Thank you!
Ralph

Plus,

I got some access log form CF, and I have noticed some of the logs are like:
“EdgeTimeToFirstByteMs”: 16301,
Other logs are like:
“EdgeTimeToFirstByteMs”: 669,

I guess this is related to the slow request, but I do not really understand what EdgeTimeToFirstByteMs means.

Thank you!
Ralph

Hi,

I resolved it.

The root cause is:

  1. Our AWS LB enabled IPv6 but did not really support it
  2. CF LB uses both IPv6/IPv4
  3. When CF LB uses IPv6, the request hangs (could not create connection with our AWS LB).
  4. I guess, there is a retry mechanism, the request success when CF LB use IPv4, and failed if every retry was send by IPv6.

The CF DNS use IPv4 only as based on my observation, I thought CF LB also do so.

The solution is: disable Dualstack of our AWS LB, use only IPv4.

Thank you.