Canceled by remote with error code 0

A significant percentage of my incoming requests will fail with “Canceled by remote with error code 0”. After several hours, my tunnel will restart with “Initiating graceful shutdown due to signal terminated”

After restarting, these errors will be completely gone for several hours. The errors will gradually increase in frequency until my tunnel once again restarts with “Initiating graceful shutdown due to signal terminated”

I’m running the latest docker image (published 5 days ago).

Short excerpt of my logs below.

2023-06-24T22:30:20Z ERR Request failed error="stream 132065 canceled by remote with error code 0" connIndex=3 dest=https://redacted.com event=0 ip=xxx.xxx.xxx.xxx type=http
2023-06-24T22:31:05Z ERR  error="stream 132081 canceled by remote with error code 0" cfRay=7dc868cefde989b0-SIN event=1 originService=http://nginx
2023-06-24T22:31:05Z ERR Request failed error="stream 132081 canceled by remote with error code 0" connIndex=3 dest=https://redacted.com event=0 ip=xxx.xxx.xxx.xxx type=http
2023-06-24T22:31:08Z ERR  error="stream 132093 canceled by remote with error code 0" cfRay=7dc868dd0d523806-IAD event=1 originService=http://nginx
2023-06-24T22:31:08Z ERR Request failed error="stream 132093 canceled by remote with error code 0" connIndex=3 dest=https://redacted.com event=0 ip=xxx.xxx.xxx.xxx type=http
2023-06-24T22:32:54Z INF Initiating graceful shutdown due to signal terminated ...
2023-06-24T22:32:55Z INF Unregistered tunnel connection connIndex=2 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:32:55Z ERR Failed to serve quic connection error="context canceled" connIndex=2 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:32:55Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:32:55Z INF Unregistered tunnel connection connIndex=0 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:32:55Z ERR Failed to serve quic connection error="context canceled" connIndex=0 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:32:55Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:32:55Z INF Unregistered tunnel connection connIndex=1 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:32:55Z ERR Failed to serve quic connection error="context canceled" connIndex=1 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:32:55Z INF Retrying connection in up to 1s connIndex=1 event=0 ip=xxx.xxx.xxx.xxx
2023-06-24T22:33:06Z INF Starting tunnel tunnelID=2f6e1df0-9b2e-4eda-b5ce-e44a9c55eb95
2023-06-24T22:33:06Z INF Version 2023.6.1
2023-06-24T22:33:06Z INF GOOS: linux, GOVersion: go1.19.10, GoArch: amd64
2023-06-24T22:33:06Z INF Settings: map[no-autoupdate:true token:*****]
2023-06-24T22:33:06Z INF Generated Connector ID: 3f68748d-dc6d-420b-8c17-143063c39561
2023-06-24T22:33:06Z INF Initial protocol quic
2023-06-24T22:33:06Z INF ICMP proxy will use 172.18.0.6 as source for IPv4
2023-06-24T22:33:06Z INF ICMP proxy will use :: as source for IPv6
2023-06-24T22:33:06Z INF Starting metrics server on 127.0.0.1:41563/metrics
2023/06/24 22:33:06 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details.

Seeing something similar with the last few versions (up to 2024.2.0), I get periodic spikes of stream 233522497 cancelled by remote with error code 0 at certain times of day, and the number of requests and response codes reported in the cloudflared metrics dips down while these are spiking. The tunnel does not restart though.
I’ve tried playing around with the go GC settings to slow down any GC in this period, but it doesn’t make any difference. And I doubt that is the cause, because it only happens at certain times of day, and the cloudflared gc metrics don’t show any unusual activity at these times.
We are getting this with requests from multiple countries, so my best guess based on the text of that error message is that there is something in the cloudflare infrastructure that causes an increased rate of client disconnects. In my more recent case at certain times of day.

By the way, I have set net.core.rmem_max and net.core.wmem_max to larger values to avoid that failed to sufficiently increase receive buffer size you are getting. Maybe that has something to do with your restarts.

I have the same logs showing.