Argo tunnel connections flaky/keeps dropping

Every time I list the tunnel via the command: cloudflared tunnel list, I see connections getting established and then drop. In other words, connections are constantly getting dropped and re-established. The end result is the internal applications/services behave sporadically

What’s causing this behavior and how to resolve it?

Edit: Added additional screenshot with seg fault:

Connection Errors/Re-trys:

Connections reestablished:

Thanks.
-r

What does the tunnel logs show?

1 Like

Where are the log files stored? I looked at the config.yml and didn’t see any variable for logfile.

cloudflared will log to the standard output, which depends on how you are executing it

you can also send its output to a log file as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/arguments#logfile

1 Like

Please see additional screenshot to the initial post for a core dump when trying to delete tunnel (cloudflared tunnel delete

As you see can see from several additional snapshots added to the original post, it seems that for some odd reasons connections are dropped and then argo attempts to reestablish connetion…After several attempts the connections are eventually established just to be dropped off…

Argo Tunnels establishes a persistent HTTP/2 connection between Cloudflare edge server and your origin server’s cloudflared daemon. It has a heartbeat check sent via origin server’s cloudflared via a HTTP/2 ping frame over port 7844. If the persistent connection between them is disrupted or broken, then a reconnection is attempted. This is evidenced in your screenshot (BTW copy and pasted text in code tags would be easier) with port 7844 tcp errors.

You just have to figure out which end or if it’s both - cloudflare or your origin server’s cloudflared side is disrupting the persistent connection. i.e. Improper or incomplete origin side outbound firewall configuration for cloudflared IPs that listen on port 7844

1 Like

Additional insights. Timeouts even when creating new tunnel. Three times it timed-out and then 4th time the tunnel was created successfully.

Since the cloudflared daemon initiates and outbound connection, there’s no firewall (pfsense) related configuration that needs to be configured. That is, all outbound connections have stateful inbound connectivity. For testing purposes, I have set the firewall optimization set to Conservative which allows idle connections from getting dropped.

If you are still seeing those “DialContext error: dial tcp 198.41.192.107:7844: i/o timeout” errors, could you please provide a traceroute or MTR?

Additionally, can you confirm your docker network interface is able to access the ports needed for tunnel as well?

If you prefer to keep some of these details private, please just submit a ticket and provide that ticket number here.