Connection with edge closed

I have 2 Cloudflared hosts that we use for a WARP tunnel that are intermittently dropping connection to the edge. I have one configured for http2:

Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z INF Lost connection with the edge connIndex=3
Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z WRN Serve tunnel error error=“connection with edge closed” connIndex=3 event=0 ip=198.41.192.227
Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z INF Retrying connection in up to 1s connIndex=3 event=0 ip=198.41.192.227
Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z INF Unregistered tunnel connection connIndex=3 event=0 ip=198.41.192.227
Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z INF Lost connection with the edge connIndex=1
Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z WRN Serve tunnel error error=“connection with edge closed” connIndex=1 event=0 ip=198.41.192.57
Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z INF Retrying connection in up to 1s connIndex=1 event=0 ip=198.41.192.57
Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z INF Unregistered tunnel connection connIndex=1 event=0 ip=198.41.192.57
Apr 10 20:01:14 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:14Z WRN Connection terminated error=“connection with edge closed” connIndex=1
Apr 10 20:01:15 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:15Z WRN Connection terminated error=“connection with edge closed” connIndex=3
Apr 10 20:01:20 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:20Z INF Registered tunnel connection connIndex=1 connection=4e1f3677-fae5-4962-93a2-88397c5f2193 event=0 ip=198.41.192.57 location=sjc06 protocol=http2
Apr 10 20:01:20 ip-172-22-54-14.us-west-1.compute.internal cloudflared[12763]: 2024-04-10T20:01:20Z INF Registered tunnel connection connIndex=3 connection=3c6e00e0-d858-43ba-9a72-abd9a5ea2cc3 event=0 ip=198.41.192.227 location=sjc06 protocol=http2

And the other for quic:

Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z INF Unregistered tunnel connection connIndex=2 event=0 ip=198.41.200.73
Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z WRN Failed to serve quic connection error=“timeout: no recent network activity” connIndex=2 event=0 ip=198.41.200.73
Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z WRN Serve tunnel error error=“timeout: no recent network activity” connIndex=2 event=0 ip=198.41.200.73
Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=198.41.200.73
Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z WRN Connection terminated error=“timeout: no recent network activity” connIndex=2
Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z INF Unregistered tunnel connection connIndex=1 event=0 ip=198.41.200.33
Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z WRN Failed to serve quic connection error=“timeout: no recent network activity” connIndex=1 event=0 ip=198.41.200.33
Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z WRN Serve tunnel error error=“timeout: no recent network activity” connIndex=1 event=0 ip=198.41.200.33
Apr 10 20:06:49 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:49Z INF Retrying connection in up to 1s connIndex=1 event=0 ip=198.41.200.33
Apr 10 20:06:50 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:06:50Z WRN Connection terminated error=“timeout: no recent network activity” connIndex=1
Apr 10 20:07:05 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:07:05Z INF Registered tunnel connection connIndex=2 connection=95b9b4db-db92-46ba-81e0-4e2e99f08406 event=0 ip=198.41.200.13 location=lax06 protocol=quic
Apr 10 20:07:05 ip-172-22-54-30.us-west-1.compute.internal cloudflared[13251]: 2024-04-10T20:07:05Z INF Registered tunnel connection connIndex=1 connection=5a47fd9c-84a5-4862-8408-0d9ebedcc366 event=0 ip=198.41.200.193 location=lax06 protocol=quic

If I restart them everything is happy for 5-15 minutes and then the errors above start to loop. A am able to connect to applications on the other side of my tunnel between error cycles.

Anyone have any idea how to troubleshoot?

Not seeing that exactly, but seeing random disconnects starting today :

{"level":"debug","time":"2024-04-10T22:45:53Z","message":"rpc rx error: timeout: no recent network activity"}
{"level":"info","event":0,"connIndex":1,"ip":"198.41.200.13","time":"2024-04-10T22:45:53Z","message":"Unregistered tunnel connection"}
{"level":"warn","event":0,"ip":"198.41.200.13","connIndex":1,"error":"timeout: no recent network activity","time":"2024-04-10T22:45:53Z","message":"Failed to serve quic connection"}
{"level":"error","time":"2024-04-10T22:45:53Z","message":"writing call: timeout: no recent network activity"}
{"level":"warn","event":0,"ip":"198.41.200.13","connIndex":1,"error":"timeout: no recent network activity","time":"2024-04-10T22:45:53Z","message":"Serve tunnel error"}
{"level":"debug","connIndex":1,"event":0,"ip":"198.41.200.113","available":16,"time":"2024-04-10T22:45:53Z","message":"edge discovery: giving new address to connection"}
{"level":"info","event":0,"ip":"198.41.200.13","connIndex":1,"time":"2024-04-10T22:45:53Z","message":"Retrying connection in up to 1s"}
{"level":"warn","error":"timeout: no recent network activity","connIndex":1,"time":"2024-04-10T22:45:54Z","message":"Connection terminated"}
{"level":"debug","time":"2024-04-10T22:45:54Z","message":"rpc rx error: timeout: no recent network activity"}
{"level":"info","event":0,"connIndex":3,"ip":"198.41.200.193","time":"2024-04-10T22:45:54Z","message":"Unregistered tunnel connection"}
{"level":"error","event":1,"connIndex":3,"originService":"http://localhost","ingressRule":0,"error":"timeout: no recent network activity","time":"2024-04-10T22:45:54Z"}
{"level":"error","event":1,"connIndex":3,"originService":"http://localhost","ingressRule":0,"error":"timeout: no recent network activity","time":"2024-04-10T22:45:54Z"}
{"level":"warn","event":0,"ip":"198.41.200.193","connIndex":3,"error":"timeout: no recent network activity","time":"2024-04-10T22:45:54Z","message":"Failed to serve quic connection"}