Cloudflare Tunnel with QUIC protocol stopped working

We updated the cloudflared client to version 2022.2.0 and the QUIC protocol stopped working. It had been working since the day the UDP requests (Private DNS) through the network tunnel were announced available. We double checked our firewall rules and tested from several OS’s and Docker images.

We finally got UDP DNS requests working once again with the December versions of the cloudflared agent. It is still reporting the same error message: “failed to dial to edge: no recent network activity”. The difference is that the older version will fallback to http2 and the February versions will retry with QUIC several times and then fail.

My understanding is that http2 would only allow TCP, so I don’t understand why the older version allow UDP when they fallback to http2.

Anyone else have issues with QUIC and the newer agents?

Hello @mschad ,

This is the first we’re hearing this feedback. I can confirm we have automated acceptance tests (i.e., running against the real infrastructure, as a real user) and using QUIC (and UDP proxying) fine, alongside with other customers.

Indeed, we made a change so that if you pick a protocol (such as quic), we no longer fallback upon connection failure (from cloudflared to our edge) so that the Tunnel does not unexpectedly lose some abilities (e.g. to proxy UDP, in the case of falling back from QUIC).

It is really impossible to have UDP proxying (and hence UDP DNS requests forwarded to the Tunnel) without QUIC. If your Tunnel was unable to connect with QUIC to our edge, then that functionality would be lost, regardless of whether cloudflared fallsback to http2 or not.

1 Like

But there is definitely something strange going on. When we start the tunnel we see that QUIC is failing and the agent switches to http2. However, our private DNS does respond even if http2 is used. I’m pasting the command line output below. I copied this using an RDP connection to a private DNS name, so I’m 100% certain that it is working. I can also access private network websites. This is all with the older versions of cloudflared.

The more important issue, though, is why does the current version fail to dial to the edge? There are no firewall rules blocking UDP out and TCP works fine. UDP fails from work networks and my home network with and without firewall.

C:\Program Files\Cloudflare>cloudflared-2021-12-1.exe tunnel --config "C:\Program Files\Cloudflare\config.yml" --protocol quic run
2022-02-15T19:17:34Z INF Starting tunnel tunnelID=8a6b94c8-107d-4e16-9a30-921c81934005
2022-02-15T19:17:34Z INF Version 2021.12.1
2022-02-15T19:17:34Z INF GOOS: windows, GOVersion: devel +a84af465cb Mon Aug 9 10:31:00 2021 -0700, GoArch: amd64
2022-02-15T19:17:34Z INF Settings: map[config:C:\Program Files\Cloudflare\config.yml cred-file:C:\Program Files\Cloudflare\8a6b94c8-107d-4e16-9a30-921c81934005.json credentials-file:C:\Program Files\Cloudflare\8a6b94c8-107d-4e16-9a30-921c81934005.json p:quic protocol:quic]
2022-02-15T19:17:34Z INF cloudflared will not automatically update on Windows systems.
2022-02-15T19:17:34Z INF Generated Connector ID: 8c98b182-5ae8-4962-bc4e-771db49aedd3
2022-02-15T19:17:34Z INF Warp-routing is enabled
2022-02-15T19:17:34Z INF Initial protocol quic
2022-02-15T19:17:34Z INF Starting metrics server on 127.0.0.1:64826/metrics
2022-02-15T19:17:34Z INF cloudflared does not support loading the system root certificate pool on Windows. Please use --origin-ca-pool <PATH> to specify the path to the certificate pool
2022-02-15T19:17:34Z WRN Your version 2021.12.1 is outdated. We recommend upgrading it to 2022.2.0
2022-02-15T19:17:39Z ERR Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-02-15T19:17:39Z ERR Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-02-15T19:17:39Z INF Retrying connection in up to 2s seconds connIndex=0
2022-02-15T19:17:40Z INF Switching to fallback protocol http2 connIndex=0
2022-02-15T19:17:41Z INF Connection a2a1c1f4-2feb-4886-948c-e95f4eab160b registered connIndex=0 location=TPA
2022-02-15T19:17:41Z INF Connection 11137291-2a5d-44bd-92ff-805a172e3900 registered connIndex=1 location=IAD
2022-02-15T19:17:43Z INF Connection ddbd87d2-98a7-43f3-8344-b098723b3097 registered connIndex=3 location=IAD
2022/02/15 15:17:44 rpc: abort: rpc: aborted by remote: rpc: shutdown
2022-02-15T19:17:47Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=2
2022-02-15T19:17:47Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=2
2022-02-15T19:17:47Z INF Retrying connection in up to 2s seconds connIndex=2
2022-02-15T19:17:48Z INF Switching to fallback protocol http2 connIndex=2
2022-02-15T19:17:48Z INF Connection f439edfc-b1db-4154-90d2-fb5ce635a289 registered connIndex=2 location=TPA

That’s because you are using 2021.12.1
If you update to the most recent version, it won’t fallback if the protocol is specified and errors occur.

The logs shown only evidence 2 connections falling back to http2. Maybe the others are still with QUIC and hence why UDP packets get through?
If you run with loglevel: debug you will be able to confirm the UDP sessions that arrive to cloudflared (via QUIC).

That’s harder for me to diagnose. In particular because the logs do not show all connections falling back.
I think it’d be easier to check this further if you update cloudflared and show the logs with debug level.

Hi,
I’m attaching a few minutes worth of the debug log with version 2022.2.0. This time it looks like 3 out of 4 connections where successful (if I understand the log correctly). This is something that was not happening for about a week prior to today.


C:\Program Files\Cloudflare>cloudflared.exe tunnel --config "C:\Program Files\Cloudflare\config.yml" --protocol quic --loglevel debug run
2022-02-17T13:31:20Z DBG Loading configuration from C:\Program Files\Cloudflare\config.yml
2022-02-17T13:31:20Z INF Starting tunnel tunnelID=8a6b94c8-107d-4e16-9a30-921c81934005
2022-02-17T13:31:20Z INF Version 2022.2.0
2022-02-17T13:31:20Z INF GOOS: windows, GOVersion: go1.17.5, GoArch: amd64
2022-02-17T13:31:20Z INF Settings: map[config:C:\Program Files\Cloudflare\config.yml cred-file:C:\Program Files\Cloudflare\8a6b94c8-107d-4e16-9a30-921c81934005.json credentials-file:C:\Program Files\Cloudflare\8a6b94c8-107d-4e16-9a30-921c81934005.json loglevel:debug p:quic protocol:quic]
2022-02-17T13:31:20Z INF cloudflared will not automatically update on Windows systems.
2022-02-17T13:31:20Z INF Generated Connector ID: 8776ce35-6a62-492c-9f81-d5ee7922c7bc
2022-02-17T13:31:20Z INF Warp-routing is enabled
2022-02-17T13:31:20Z INF Initial protocol quic
2022-02-17T13:31:20Z INF Starting metrics server on 127.0.0.1:58571/metrics
2022-02-17T13:31:20Z INF cloudflared does not support loading the system root certificate pool on Windows. Please use --origin-ca-pool <PATH> to specify the path to the certificate pool
2022-02-17T13:31:20Z DBG looking up edge SRV record domain=_origintunneld._tcp.argotunnel.com
2022-02-17T13:31:20Z DBG edgediscovery - GetAddr: Giving connection its new address connIndex=0
2022-02-17T13:31:21Z DBG rpcconnect: tx (bootstrap = (questionId = 0, deprecatedObjectId = <opaque pointer>))
2022-02-17T13:31:21Z DBG rpcconnect: tx (call = (questionId = 1, target = (promisedAnswer = (questionId = 0, transform = [ ])), interfaceId = 17804583019846587543, methodId = 0, allowThirdPartyTailCall = false, params = (content = <opaque pointer>, capTable = []), sendResultsTo = (caller = void)))
2022-02-17T13:31:21Z DBG rpcconnect: rx (return = (answerId = 0, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [(senderHosted = 0)])))
2022-02-17T13:31:21Z DBG rpcconnect: tx (finish = (questionId = 0, releaseResultCaps = false))
2022-02-17T13:31:21Z DBG rpcconnect: rx (return = (answerId = 1, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [])))
2022-02-17T13:31:21Z DBG rpcconnect: tx (finish = (questionId = 1, releaseResultCaps = false))
2022-02-17T13:31:21Z INF Connection 0a883ed1-0835-42ef-b06f-26e60e015ff4 registered connIndex=0 location=ATL
2022-02-17T13:31:21Z DBG edgediscovery - GetDifferentAddr: Giving connection its new address connIndex=1
2022-02-17T13:31:21Z DBG rpcconnect: tx (bootstrap = (questionId = 0, deprecatedObjectId = <opaque pointer>))
2022-02-17T13:31:21Z DBG rpcconnect: tx (call = (questionId = 1, target = (promisedAnswer = (questionId = 0, transform = [])), interfaceId = 17804583019846587543, methodId = 0, allowThirdPartyTailCall = false, params = (content = <opaque pointer>, capTable = []), sendResultsTo = (caller = void)))
2022-02-17T13:31:22Z DBG rpcconnect: rx (return = (answerId = 0, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [(senderHosted = 0)])))
2022-02-17T13:31:22Z DBG rpcconnect: tx (finish = (questionId = 0, releaseResultCaps = false))
2022-02-17T13:31:22Z DBG rpcconnect: rx (return = (answerId = 1, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [])))
2022-02-17T13:31:22Z INF Connection 3fad465d-4f53-4e2a-8bba-8f479abeb82e registered connIndex=1 location=IAD
2022-02-17T13:31:22Z DBG rpcconnect: tx (finish = (questionId = 1, releaseResultCaps = false))
2022-02-17T13:31:22Z DBG edgediscovery - GetDifferentAddr: Giving connection its new address connIndex=2
2022-02-17T13:31:23Z DBG edgediscovery - GetDifferentAddr: Giving connection its new address connIndex=3
2022-02-17T13:31:24Z DBG rpcconnect: tx (bootstrap = (questionId = 0, deprecatedObjectId = <opaque pointer>))
2022-02-17T13:31:24Z DBG rpcconnect: tx (call = (questionId = 1, target = (promisedAnswer = (questionId = 0, transform = [])), interfaceId = 17804583019846587543, methodId = 0, allowThirdPartyTailCall = false, params = (content = <opaque pointer>, capTable = []), sendResultsTo = (caller = void)))
2022-02-17T13:31:24Z DBG rpcconnect: rx (return = (answerId = 0, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [(senderHosted = 0)])))
2022-02-17T13:31:24Z DBG rpcconnect: tx (finish = (questionId = 0, releaseResultCaps = false))
2022-02-17T13:31:24Z DBG rpcconnect: rx (return = (answerId = 1, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [])))
2022-02-17T13:31:24Z DBG rpcconnect: tx (finish = (questionId = 1, releaseResultCaps = false))
2022-02-17T13:31:24Z INF Connection 5bbb465f-2623-4399-9f27-38337f62934a registered connIndex=3 location=IAD
2022-02-17T13:31:27Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=2
2022-02-17T13:31:27Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=2
2022-02-17T13:31:27Z INF Retrying connection in up to 2s seconds connIndex=2
2022-02-17T13:31:34Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=2
2022-02-17T13:31:34Z DBG Registered session 0ca0ad61-ff25-4abf-8747-2364d4c40c2d, 192.168.1.255, 57621 connIndex=0
2022-02-17T13:31:34Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=2
2022-02-17T13:31:34Z INF Retrying connection in up to 4s seconds connIndex=2
2022-02-17T13:31:37Z DBG rpcconnect: tx (bootstrap = (questionId = 0, deprecatedObjectId = <opaque pointer>))
2022-02-17T13:31:37Z DBG rpcconnect: tx (call = (questionId = 1, target = (promisedAnswer = (questionId = 0, transform = [])), interfaceId = 17804583019846587543, methodId = 0, allowThirdPartyTailCall = false, params = (content = <opaque pointer>, capTable = []), sendResultsTo = (caller = void)))
2022-02-17T13:31:37Z DBG rpcconnect: rx (return = (answerId = 0, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [(senderHosted = 0)])))
2022-02-17T13:31:37Z DBG rpcconnect: tx (finish = (questionId = 0, releaseResultCaps = false))
2022-02-17T13:31:37Z DBG rpcconnect: rx (return = (answerId = 1, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [])))
2022-02-17T13:31:37Z INF Connection 5accf61a-64b1-413e-ab8f-51ab52d119ac registered connIndex=2 location=ATL
2022-02-17T13:31:37Z DBG rpcconnect: tx (finish = (questionId = 1, releaseResultCaps = false))
2022-02-17T13:31:39Z DBG Registered session 79663045-ada3-47e7-ad5b-b9769b725b44, 192.168.1.255, 17500 connIndex=0
2022-02-17T13:31:39Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=0 sessionID=0ca0ad61-ff25-4abf-8747-2364d4c40c2d
2022-02-17T13:31:39Z DBG Registered session 0ca0ad61-ff25-4abf-8747-2364d4c40c2d, 192.168.1.255, 57621 connIndex=1
2022-02-17T13:31:44Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=0 sessionID=79663045-ada3-47e7-ad5b-b9769b725b44
2022-02-17T13:31:44Z DBG Registered session 79663045-ada3-47e7-ad5b-b9769b725b44, 192.168.1.255, 17500 connIndex=1
2022-02-17T13:31:45Z DBG Registered session da78e5db-264c-4f1f-bcf5-a883f271bf9d, 192.168.1.255, 137 connIndex=0
2022-02-17T13:31:54Z DBG Registered session 11326381-1e99-478e-86b5-cd46274f3841, 192.168.1.4, 35928 connIndex=0
2022-02-17T13:32:04Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=1 sessionID=0ca0ad61-ff25-4abf-8747-2364d4c40c2d
2022-02-17T13:32:04Z DBG Registered session 153a5ba1-9c43-428d-8d71-e8726fc47006, 192.168.1.255, 57621 connIndex=0
2022-02-17T13:32:09Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=1 sessionID=79663045-ada3-47e7-ad5b-b9769b725b44
2022-02-17T13:32:09Z DBG Registered session a557d43c-5688-4104-8a3f-a23a7528c072, 192.168.1.255, 17500 connIndex=0
2022-02-17T13:32:16Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=0 sessionID=da78e5db-264c-4f1f-bcf5-a883f271bf9d
2022-02-17T13:32:24Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=0 sessionID=11326381-1e99-478e-86b5-cd46274f3841
2022-02-17T13:32:34Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=0 sessionID=153a5ba1-9c43-428d-8d71-e8726fc47006
2022-02-17T13:32:34Z DBG Registered session 3dcef34e-7f03-4c99-8b59-5cedaac47ec9, 192.168.1.255, 57621 connIndex=0
2022-02-17T13:32:39Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=0 sessionID=a557d43c-5688-4104-8a3f-a23a7528c072
2022-02-17T13:32:39Z DBG Registered session 1daca829-bba2-45da-b99f-15b0f742f405, 192.168.1.255, 17500 connIndex=0
2022-02-17T13:33:04Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=0 sessionID=3dcef34e-7f03-4c99-8b59-5cedaac47ec9
2022-02-17T13:33:04Z DBG Registered session b7c58a6b-7694-492d-9f03-e202e958d03d, 192.168.1.255, 57621 connIndex=0
2022-02-17T13:33:09Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=0 sessionID=1daca829-bba2-45da-b99f-15b0f742f405
2022-02-17T13:33:10Z DBG Registered session e6778683-3bf6-46dc-bb3e-25fd9bec9029, 192.168.1.255, 17500 connIndex=0

I can’t explain out of the box why you have some hardship in connecting to our edge with QUIC, but it seems to work for most of the times. Anyway thanks to the 4 connections, at least 1 is always up, and you are getting UDP traffic. So it seems that from a user perspective, it should be fine.

INF Starting metrics server on 127.0.0.1:43867/metrics
failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See

Maybe is this issue [ UDP Receive Buffer Size]
[UDP Receive Buffer Size · lucas-clemente/quic-go Wiki (github)]

Google search
lucas-clemente WRN Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity

I can’t trouble shoot what port is Lucas-clemente quic is using, when I totally turn off the firewall, the warning won’t happen