Resolved: Connection Issues with Cloudflared due to Ingress UDP Traffic

Hello everyone,

I wanted to share a solution to a problem I faced with cloudflared running in a Kubernetes environment on Hetzner. The issue was that cloudflared had trouble establishing a connection initially, but eventually managed to connect using the http2 protocol after some time. Here are the relevant logs:

2023-05-28T14:19:30Z INF Starting tunnel tunnelID=[EDITED]
2023-05-28T14:19:30Z INF Version 2023.5.1
2023-05-28T14:19:30Z INF GOOS: linux, GOVersion: go1.19.9, GoArch: amd64
2023-05-28T14:19:30Z INF Settings: map[edge-ip-version:4 no-autoupdate:true token:*****]
2023-05-28T14:19:30Z INF Environmental variables map[TUNNEL_METRICS:0.0.0.0:2000]
2023-05-28T14:19:30Z INF Generated Connector ID: [EDITED]
2023-05-28T14:19:30Z INF Initial protocol quic
2023-05-28T14:19:30Z INF ICMP proxy will use [EDITED] as source for IPv4
2023-05-28T14:19:30Z INF ICMP proxy will use [EDITED] in zone eth0 as source for IPv6
2023-05-28T14:19:30Z INF Starting metrics server on [::]:2000/metrics
2023-05-28T14:19:35Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.77
2023-05-28T14:19:35Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.192.77
2023-05-28T14:22:33Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not work unless your cloudflared can connect with Cloudflare Network with `quic`.
2023-05-28T14:22:33Z INF Switching to fallback protocol http2 connIndex=0 event=0 ip=198.41.192.167
2023-05-28T14:22:48Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.200.63:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.63
2023-05-28T14:22:48Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.200.63:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.63
2023-05-28T14:22:48Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.63
2023-05-28T14:22:50Z INF Registered tunnel connection connIndex=0 connection=[EDITED] event=0 ip=198.41.200.23 location=DME protocol=http2

Initially, I suspected that the problem was related to blocked ingress UDP traffic, as the service tried to connect via QUIC on port 7844, which wasn’t working, while connections on port 80 were fine.

After some troubleshooting, I found that allowing ingress traffic from port 8443/udp in my Hetzner firewall resolved the issue. This was likely necessary as Hetzner’s firewall is not stateful, and with UFW only it worked perfectly fine.

I wanted to share this experience as I haven’t found any documentation mentioning the need to allow ingress traffic for cloudflared to work properly. I hope this helps anyone else facing a similar issue.

And to the Cloudflare team, it might be helpful to include a note about this in the cloudflared documentation to guide users in the future.

Cheers!

1 Like

Opening up UDP port on Hetzner firewall worked for me also. Thanks!

(their is nothing in the docs about this)

How should I configure my firewall to improve UDP traffic issues? thanks

This worked for me:

    Log on to Hetzner Robot and select Firewall for your server
    Add a new rule, name it something appropriate like UDP Established v4
    Set the version to ipv4 and the protocol to UDP.
    Set the destination port to 32768-65535
    Set the action to Accept
    Repeat the above for ipv6 if your server supports it
    Hit “Save” at the bottom of the page

Source:https://stretton.xyz/2023/05/31/hetzner-cloudflare-tunnel-connection-refused/