504 response from cloudflare when backend takes > 15 seconds to respond

My current setup is using docker swarm, exposing port 80 to the internet (with cloudflare doing flexible TLS in front of).

Inside docker swarm, it is configured as such:
Loadbalancer: Envoy virtual host routing to node js apps based on url.
App server 1: Node app listening on port 3000
App server 2: Node app listening on port 3000

With cloudflare in front of docker swarm, I have a 40% chance of getting HTTP 504 response.

. The 504 is always returned after 15 seconds. Without cloudflare enabled, it works correctly.

I saw this post (timeout - Is it possible to increase CloudFlare time-out? - Stack Overflow) which suggests that envoy doesn’t TCP handshake in time. I tested locally, I do see TCP handshakes immediately. Any thoughts on what may be wrong?

Issue is envoy timeout on backend. Solution is to increase timeout, like this example: https://gist.githubusercontent.com/hinawatts/d21d637f225024c1daeb3b9c1c1d5025/raw/844539ae652eaf17a12c4bcbce02303b34ef9a91/envoy-route.yaml