That shouldn’t be related. The default timeout with a 524 only applies when the connection has been already established and when the origin does not respond. In your case it seems the TCP connection itself didn’t work.
When you experience the timeouts, can you ping that host?
Also, you seem to be using IPv6, could you check if you ever experience timeouts as well when you force IPv4?