High latency with Cloudflare tunnel

[kun.yang]

What is the name of the domain?

auth.agnos.dev

What is the issue you’re encountering

Randomly high latency with Cloudflare tunnel

What steps have you taken to resolve the issue?

Use Cloudflare proxied A/AAAA record to connect my origin server directly,
with it I have a stable latency around 50ms(measured with AWS server in the nearest region)

What are the steps to reproduce the issue?

Use Cloudflare tunnel to connect Cloudflare with origin server,
and it will cause high (500ms ~ 1500ms) + unstable latency.

Have tried set TUNNEL_EDGE_IP_VERSION to 4 or 6,
and TUNNEL_TRANSPORT_PROTOCOL to http2 or quic,
none of it work.

My origin server itself have a very low latency to Cloudflare front

ping -c5 auth.agnos.dev
PING auth.agnos.dev(2606:4700:3037::ac43:d158 (2606:4700:3037::ac43:d158)) 56 data bytes
64 bytes from 2606:4700:3037::ac43:d158 (2606:4700:3037::ac43:d158): icmp_seq=1 ttl=53 time=4.30 ms
64 bytes from 2606:4700:3037::ac43:d158 (2606:4700:3037::ac43:d158): icmp_seq=2 ttl=53 time=4.57 ms
64 bytes from 2606:4700:3037::ac43:d158 (2606:4700:3037::ac43:d158): icmp_seq=3 ttl=53 time=5.48 ms
64 bytes from 2606:4700:3037::ac43:d158 (2606:4700:3037::ac43:d158): icmp_seq=4 ttl=53 time=7.33 ms
64 bytes from 2606:4700:3037::ac43:d158 (2606:4700:3037::ac43:d158): icmp_seq=5 ttl=53 time=5.79 ms

Screenshot of the error

[Erisa]

Maybe your ISP/cloud provider hates QUIC? You can try adding --protocol http2 to your cloudflared commandline and see if that helps or hinders. On a systemd setup this would be done using sudo systemctl edit --full cloudflared

[kun.yang]

Thanks for your reply,

I have test cloudflare tunnel with HTTP2, nothing changes.

After that I did a further monitoring, and found that the high latency may not caused by the tunnel, but the cloudflare itself.

Here I did a comparison with AWS, Cloudflare, Cloudflare with tunnel(HTTP2), with same origin server.

AWS as front

Cloudflare proxied

Cloudflare with Tunnel(HTTP2)

As we can see, there is certain high latency period with cloudflare, no matter was it tunneled or not.

I have no idea why those high latency period comes from. May cloudflare routing my packets to a wrong region? or just want give me some latency with no reason since I am using the free plan?

[kun.yang]

Seems like the high latency period always happens in 21:30 ~ 0:30.

Is cloudflare doing prime time control to throttle the performance of “non-prime” user?
Would upgrading to “Pro/Business” plan solve this?
I am encountering this high ping period almost every day prime time, quite disgusting.

Anyone know anything about this?
Considering to switch to AWS now if there is no solution for me to solve this problem

[kun.yang]