Since upgrading to Cloudflared version 2022.5.0, I’m getting intermittent failures on my private routing tunnel. Below is the info from my log. Nothing has changed in my environment except the upgrade to the Cloudflared.
I am too having this exact same issue from last few days. Tunnels suddenly stops working with below errors,
May 10 21:35:27 Ubuntu cloudflared[56025]: 2022-05-10T16:05:27Z INF Starting tunnel tunnelID=d63da755-a723-4396-836f-948e2c6480c1
May 10 21:35:27 Ubuntu cloudflared[56025]: 2022-05-10T16:05:27Z INF Version 2022.5.0
May 10 21:35:27 Ubuntu cloudflared[56025]: 2022-05-10T16:05:27Z INF GOOS: linux, GOVersion: go1.17.5, GoArch: amd64
May 10 21:35:27 Ubuntu cloudflared[56025]: 2022-05-10T16:05:27Z INF Settings: map[config:/etc/cloudflared/config.yml cred-file:/home/santosh/.cloudflared/d63da75>
May 10 21:35:27 Ubuntu cloudflared[56025]: 2022-05-10T16:05:27Z INF Generated Connector ID: ca834f77-bddc-4fb6-b0e4-cc34da9b5916
May 10 21:35:27 Ubuntu cloudflared[56025]: 2022-05-10T16:05:27Z INF Initial protocol quic
May 10 21:35:27 Ubuntu cloudflared[56025]: 2022-05-10T16:05:27Z INF Starting metrics server on 127.0.0.1:42935/metrics
May 10 21:35:28 Ubuntu cloudflared[56025]: 2022/05/10 21:35:28 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB)>
May 10 21:35:29 Ubuntu cloudflared[56025]: 2022-05-10T16:05:29Z INF Connection 8c84d983-f0f1-4ea6-98cf-<> registered connIndex=0 location=DEL
May 10 21:35:29 Ubuntu systemd[1]: Started cloudflared.
May 10 21:35:29 Ubuntu cloudflared[56025]: 2022-05-10T16:05:29Z INF Connection c357733b-f545-4157-bf41-026fe1bfb7e6 registered connIndex=1 location=BOM
May 10 21:35:30 Ubuntu cloudflared[56025]: 2022-05-10T16:05:30Z INF Connection da119a52-8eab-455a<> registered connIndex=2 location=DEL
May 10 21:35:31 Ubuntu cloudflared[56025]: 2022-05-10T16:05:31Z INF Connection 797710a2-f502<> registered connIndex=3 location=BOM
May 11 01:46:15 Ubuntu cloudflared[56025]: 2022-05-10T20:16:15Z INF Unregistered tunnel connection connIndex=2
May 11 01:46:15 Ubuntu cloudflared[56025]: 2022-05-10T20:16:15Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent netwo>
May 11 01:46:15 Ubuntu cloudflared[56025]: 2022-05-10T20:16:15Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" >
May 11 01:46:15 Ubuntu cloudflared[56025]: 2022-05-10T20:16:15Z INF Retrying connection in up to 1s seconds connIndex=2
May 11 01:46:16 Ubuntu cloudflared[56025]: 2022-05-10T20:16:16Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network w>
May 11 01:46:16 Ubuntu cloudflared[56025]: If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudf>
May 11 01:46:16 Ubuntu cloudflared[56025]: 2022-05-10T20:16:16Z INF Switching to fallback protocol http2 connIndex=2
May 11 01:46:16 Ubuntu cloudflared[56025]: 2022-05-10T20:16:16Z INF Connection b31fda6c-65af-4573<> registered connIndex=2 location=DEL
May 11 04:23:48 Ubuntu cloudflared[56025]: 2022-05-10T22:53:48Z INF Unregistered tunnel connection connIndex=0
May 11 04:23:48 Ubuntu cloudflared[56025]: 2022-05-10T22:53:48Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent netwo>
May 11 04:23:48 Ubuntu cloudflared[56025]: 2022-05-10T22:53:48Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" >
May 11 04:23:48 Ubuntu cloudflared[56025]: 2022-05-10T22:53:48Z INF Retrying connection in up to 1s seconds connIndex=0
May 11 04:23:50 Ubuntu cloudflared[56025]: 2022-05-10T22:53:50Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network w>
May 11 04:23:50 Ubuntu cloudflared[56025]: If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudf>
May 11 04:23:50 Ubuntu cloudflared[56025]: 2022-05-10T22:53:50Z INF Switching to fallback protocol http2 connIndex=0
If I restart the cloudlflared service systemctl restart cloudflared then it will work for few minutes and then again will stop working.
I narrowed down the issue to our Veeam backup process and my Linux VM. Veeam Backup & Replication works in conjunction with VMware to create and delete a snapshot of a VM during the backup process. When the snapshot is deleted, disk consolidation occurs. The disk consolidation was causing the tunnel to disconnect, even though the cloudflared tunnel was still up. Would be great if the tunnel would automatically recover, but I was able to create a workaround with a crontab job to restart the cloudflared tunnel service everyday at a set time, after the backup of the effected VM has completed.