Tunnels INACTIVE - ERR Serve tunnel error error="TLS handshake with edge error..."

As of ~23-24th June 2022, I’m seeing the following in logs and no new tunnel connections are able to be established (I have one client still connected - it hasn’t rebooted, yet, so I assume it’s still using a previously established connection) - all other tunnels are showing as “INACTIVE” in Zero Trust Dashboard > Access > Tunnels

ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:58235->198.41.200.193:7844: i/o timeout" connIndex=0

I’ve not changed anything in the setup of any of my client machines - and all are running cloudflared version 2022.6.3 (built 2022-06-20T12:52:31Z)

This is happening without a firewall, from different IP locations, with or without WARP usage.

It doesn’t seem to be a tcp connection / port issue as that appears to be getting through:

$ nc -vz 198.41.200.193 7844
found 0 associations
found 1 connections:
     1:	flags=82<CONNECTED,PREFERRED>
	outif utun0
	src 172.16.0.2 port 58247
	dst 198.41.200.193 port 7844
	rank info not available
	TCP aux info available

Connection to 198.41.200.193 port 7844 [tcp/*] succeeded!

Anyone else seeing this?

By the way, my log file at /Library/Logs/com.cloudflare.cloudflared.err.log has grown to be over 400MB. I think that needs addressing too (archive, compress, etc, periodically, maybe) :thinking:
I’m manually deleting it at the moment before a reboot :flushed:

Predictably, as of today I now have 0 tunnels active…

If this is happening to just me, is there a way that I can debug this? Apart from the one machine I experimented with on the 25th, the others have been untouched config-wise (just cloudflared and WARP client updates) and all has been well until now :disappointed_relieved:

Turns out this is indeed a WARP routing issue. I didn’t think it was at first as some machines were still working with it, but maybe they still had something cached. Anyway, no sign of anything working now with WARP enabled, but all works as expected with WARP app set to only use DoH. I’ve discounted the version of the WARP App as I’ve tried the last 6 (and everything was definitely working back then), so it has to be on the backend.

Anyway, example of working:

# Application > Cloudflare WARP > Settings > "Gateway with DoH"
$ cloudflared tunnel run cloudflared-tunnel-test
2022-06-28T13:03:54Z INF Starting tunnel tunnelID=83de11d6-4c32-47cc-ba33-deadfe88adb1
2022-06-28T13:03:54Z INF Version 2022.6.3
2022-06-28T13:03:54Z INF GOOS: darwin, GOVersion: go1.18.3, GoArch: amd64
2022-06-28T13:03:54Z INF Settings: map[config:/etc/cloudflared/config.yml cred-file:/etc/cloudflared/83de11d6-4c32-47cc-ba33-deadfe88adb1.json credentials-file:/etc/cloudflared/83de11d6-4c32-47cc-ba33-deadfe88adb1.json]
2022-06-28T13:03:54Z INF Generated Connector ID: 6a043d7d-182c-4b76-9114-37e9e0c9ebc8
2022-06-28T13:03:54Z INF Initial protocol quic
2022-06-28T13:03:54Z INF Starting metrics server on 127.0.0.1:50335/metrics
2022-06-28T13:03:54Z INF Connection 0658d3b8-d8d4-4830-b891-7554bc206401 registered connIndex=0 ip=198.41.200.13 location=AMS
2022-06-28T13:03:55Z INF Connection 447e3b55-0cc6-4a92-bd7d-bf48827d7a0c registered connIndex=1 ip=198.41.192.227 location=LHR
2022-06-28T13:03:56Z INF Connection 0f8d7e3c-301c-4b4a-92e2-743e0bd1b223 registered connIndex=2 ip=198.41.200.53 location=AMS
2022-06-28T13:03:57Z INF Connection 618df006-7313-4a29-82d2-6e76c49c39fa registered connIndex=3 ip=198.41.192.107 location=LHR

Example of NOT working

# Application > Cloudflare WARP > Settings > "Gateway with WARP"
$ cloudflared tunnel run cloudflared-tunnel-test
2022-06-28T12:59:52Z INF Starting tunnel tunnelID=83de11d6-4c32-47cc-ba33-deadfe88adb1
2022-06-28T12:59:52Z INF Version 2022.6.3
2022-06-28T12:59:52Z INF GOOS: darwin, GOVersion: go1.18.3, GoArch: amd64
2022-06-28T12:59:52Z INF Settings: map[config:/etc/cloudflared/config.yml cred-file:/etc/cloudflared/83de11d6-4c32-47cc-ba33-deadfe88adb1.json credentials-file:/etc/cloudflared/83de11d6-4c32-47cc-ba33-deadfe88adb1.json]
2022-06-28T12:59:52Z INF Generated Connector ID: d1e1f542-691e-4643-ba9f-4ecd68c5c0ff
2022-06-28T12:59:52Z INF Initial protocol quic
2022-06-28T12:59:52Z INF Starting metrics server on 127.0.0.1:50224/metrics
2022-06-28T12:59:57Z ERR Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-28T12:59:57Z ERR Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-28T12:59:57Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-28T12:59:57Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-28T12:59:57Z INF Switching to fallback protocol http2 connIndex=0
.
.
.
... followed by copious 🤮 of errors to the logs every few seconds

Maybe it’s fallout from the huge Cloudflare outage from the 21st ?

For now, I’m able to access machines again if the Cloudflare WARP Application is set to just “Gateway with DoH” :roll_eyes:

Am I understanding correctly that your cloudflared tunnel daemons are running in machines that also have WARP running?

What’s your use case for that?

(I’m curious about it; in practice, it should still work, and I just tried it on my device — except that you can only get each tunnel to connect 3 times rather than 4 for reasons explained in Unable to establish connection. error="already connected to this server, trying another address" · Issue #344 · cloudflare/cloudflared · GitHub )

Yes, exactly.

The use case is on managed machines that have a user login. The WARP app provides DoH (at a minimum) and WARP to protect ingress in general. I still need a way to access them for admin and I do that via a Cloudflare tunnel. So both cloudflared and the ‘WARP App’ are both in play. Assume that’s all ‘allowed’
?

Used to work fine until this last week.

Yes, it is allowed — the only caveat is what I wrote on the previous comment.
Also, as I commented above, it works on my local setup too, I tried it today when I read your thread.

Did you change the Split Tunnel config by any chance? https://developers.cloudflare.com/cloudflare-one/connections/connect-devices/warp/exclude-traffic/split-tunnels/

Also, can you share a longer log from the “Example of NOT working” that includes more lines after the “Switching to fallback protocol”?

Can you also check your Gateway Policies (https://developers.cloudflare.com/cloudflare-one/policies/filtering/network-policies/) to make sure you’re not blocking the “cloudflared tunnel” outbound connections to Cloudflare’s Network (https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/)?

Yep, I can send better logs :nerd_face:. Is it possible to DM them or send a file somewhere?

Split Tunnels unused / default
Gateway Policies (DNS, Network, HTTP) unused / default

Thanks for looking into this.

You can paste it here like you’ve done already. Just don’t truncate so early, add like 20 more lines.

OK, thanks…

So, with tunnel “active” (connected when using “Gateway with DoH”) the following shows the logs when at 2022-06-29T09:14:00Z (approx) the service mode was changed to - “Gateway with WARP”

2022-06-29T08:31:26Z INF Connection b7e96a7f-bd66-4418-8952-6df56864faa1 registered connIndex=0 ip=198.41.200.193 location=AMS
2022-06-29T08:31:26Z INF Connection d0538742-819c-4fcd-af8a-8880759ec206 registered connIndex=1 ip=198.41.192.167 location=LHR
2022-06-29T08:31:27Z INF Connection 89610ee1-7b59-4248-8b02-aad2e0a2bb5d registered connIndex=2 ip=198.41.200.53 location=AMS
2022-06-29T08:31:28Z INF Connection 7113e562-85c9-4c0c-9e80-db394c4aed90 registered connIndex=3 ip=198.41.192.107 location=LHR
2022-06-29T09:14:13Z INF Unregistered tunnel connection connIndex=1
2022-06-29T09:14:13Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1
2022-06-29T09:14:13Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1
2022-06-29T09:14:13Z INF Retrying connection in up to 1s seconds connIndex=1
2022-06-29T09:14:13Z INF Unregistered tunnel connection connIndex=3
2022-06-29T09:14:13Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3
2022-06-29T09:14:13Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3
2022-06-29T09:14:13Z INF Retrying connection in up to 1s seconds connIndex=3
2022-06-29T09:14:13Z INF Unregistered tunnel connection connIndex=2
2022-06-29T09:14:13Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=2
2022-06-29T09:14:13Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=2
2022-06-29T09:14:13Z INF Retrying connection in up to 1s seconds connIndex=2
2022-06-29T09:14:13Z INF Unregistered tunnel connection connIndex=0
2022-06-29T09:14:13Z ERR Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0
2022-06-29T09:14:13Z ERR Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0
2022-06-29T09:14:13Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:14:14Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:14:14Z INF Switching to fallback protocol http2 connIndex=0
2022-06-29T09:14:14Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=1
2022-06-29T09:14:14Z INF Switching to fallback protocol http2 connIndex=1
2022-06-29T09:14:14Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=2
2022-06-29T09:14:14Z INF Switching to fallback protocol http2 connIndex=2
2022-06-29T09:14:14Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=3
2022-06-29T09:14:14Z INF Switching to fallback protocol http2 connIndex=3
2022-06-29T09:14:14Z ERR Unable to establish connection. error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:14:15Z INF Connection 2124b117-1d6d-493e-b2d1-241b30babdd8 registered connIndex=1 ip=172.16.0.2 location=LHR
2022-06-29T09:14:15Z WRN Connection terminated error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:14:29Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57110->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:14:29Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57110->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:14:29Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:14:29Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57112->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:14:29Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57112->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:14:29Z INF Retrying connection in up to 1s seconds connIndex=2
2022-06-29T09:14:30Z WRN Failed to serve quic connection error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:14:30Z WRN Unable to establish connection. error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:14:31Z WRN Connection terminated error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:14:43Z WRN Failed to serve quic connection error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:14:43Z WRN Unable to establish connection. error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:14:44Z WRN Connection terminated error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:14:44Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57140->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:14:44Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57140->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:14:44Z INF Retrying connection in up to 4s seconds connIndex=2
2022-06-29T09:14:45Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57141->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:14:45Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57141->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:14:45Z INF Retrying connection in up to 4s seconds connIndex=0
2022-06-29T09:15:01Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57143->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:15:01Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57143->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:15:01Z INF Retrying connection in up to 8s seconds connIndex=0
2022-06-29T09:15:03Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57144->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:15:03Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57144->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:15:03Z INF Retrying connection in up to 8s seconds connIndex=2
2022-06-29T09:15:18Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57147->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:15:18Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57147->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:15:18Z INF Retrying connection in up to 16s seconds connIndex=0
2022-06-29T09:15:23Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57148->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:15:23Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57148->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:15:23Z INF Retrying connection in up to 16s seconds connIndex=2
2022-06-29T09:15:36Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57159->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:15:36Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57159->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:15:36Z INF Retrying connection in up to 32s seconds connIndex=0
2022-06-29T09:15:43Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:15:43Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:15:43Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:15:44Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:15:44Z INF Switching to fallback protocol http2 connIndex=0
2022-06-29T09:15:48Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57162->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:15:48Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57162->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:15:48Z INF Retrying connection in up to 32s seconds connIndex=2
2022-06-29T09:15:59Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57163->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:15:59Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57163->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:15:59Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:16:02Z WRN Failed to serve quic connection error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:16:02Z WRN Unable to establish connection. error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:16:03Z WRN Connection terminated error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:16:14Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57165->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:16:14Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57165->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:16:14Z INF Retrying connection in up to 4s seconds connIndex=0
2022-06-29T09:16:16Z WRN Connection terminated error="TLS handshake with edge error: read tcp 172.16.0.2:57162->198.41.200.53:7844: i/o timeout" connIndex=2
2022-06-29T09:16:31Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57168->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:16:31Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57168->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:16:31Z INF Retrying connection in up to 8s seconds connIndex=0
2022-06-29T09:16:49Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57174->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:16:49Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57174->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:16:49Z INF Retrying connection in up to 16s seconds connIndex=0
2022-06-29T09:17:19Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57179->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:17:19Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57179->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:17:19Z INF Retrying connection in up to 32s seconds connIndex=0
2022-06-29T09:17:45Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:17:45Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:17:45Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:17:46Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:17:46Z INF Switching to fallback protocol http2 connIndex=0
2022-06-29T09:17:56Z WRN Failed to serve quic connection error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:17:56Z WRN Unable to establish connection. error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:17:57Z WRN Connection terminated error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:18:01Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=2
2022-06-29T09:18:01Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=2
2022-06-29T09:18:01Z INF Retrying connection in up to 2s seconds connIndex=2
2022-06-29T09:18:01Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=2
2022-06-29T09:18:01Z INF Switching to fallback protocol http2 connIndex=2
2022-06-29T09:18:01Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57188->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:18:01Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57188->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:18:01Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:18:16Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57198->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:18:16Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57198->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:18:16Z INF Retrying connection in up to 1s seconds connIndex=2
2022-06-29T09:18:17Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57199->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:18:17Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57199->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:18:17Z INF Retrying connection in up to 4s seconds connIndex=0
2022-06-29T09:18:21Z WRN Failed to serve quic connection error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:18:21Z WRN Unable to establish connection. error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:18:22Z WRN Connection terminated error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:18:32Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57209->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:18:32Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57209->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:18:32Z INF Retrying connection in up to 4s seconds connIndex=2
2022-06-29T09:18:34Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57210->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:18:34Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57210->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:18:34Z INF Retrying connection in up to 8s seconds connIndex=0
2022-06-29T09:18:48Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57214->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:18:48Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57214->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:18:48Z INF Retrying connection in up to 8s seconds connIndex=2
2022-06-29T09:18:56Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57222->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:18:56Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57222->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:18:56Z INF Retrying connection in up to 16s seconds connIndex=0
2022-06-29T09:19:05Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57224->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:19:05Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57224->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:19:05Z INF Retrying connection in up to 16s seconds connIndex=2
2022-06-29T09:19:06Z WRN Failed to serve quic connection error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:19:06Z WRN Unable to establish connection. error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:19:07Z WRN Connection terminated error="already connected to this server, trying another address" connIndex=3
2022-06-29T09:19:19Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57225->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:19:19Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57225->198.41.200.233:7844: i/o timeout" connIndex=0
2022-06-29T09:19:19Z INF Retrying connection in up to 32s seconds connIndex=0
2022-06-29T09:19:26Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57226->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:19:26Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57226->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:19:26Z INF Retrying connection in up to 32s seconds connIndex=2
2022-06-29T09:19:46Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:19:46Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:19:46Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:19:47Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:19:47Z INF Switching to fallback protocol http2 connIndex=0
2022-06-29T09:19:54Z WRN Connection terminated error="TLS handshake with edge error: read tcp 172.16.0.2:57226->198.41.200.113:7844: i/o timeout" connIndex=2
2022-06-29T09:20:02Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57229->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:20:02Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57229->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:20:02Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:20:19Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57230->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:20:19Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57230->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:20:19Z INF Retrying connection in up to 4s seconds connIndex=0
2022-06-29T09:20:36Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57232->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:20:36Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57232->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:20:36Z INF Retrying connection in up to 8s seconds connIndex=0
2022-06-29T09:20:58Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57235->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:20:58Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57235->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:20:58Z INF Retrying connection in up to 16s seconds connIndex=0
2022-06-29T09:21:13Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57239->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:21:13Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57239->198.41.200.13:7844: i/o timeout" connIndex=0
2022-06-29T09:21:13Z INF Retrying connection in up to 32s seconds connIndex=0
2022-06-29T09:21:27Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:21:27Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:21:27Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:21:28Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:21:28Z INF Switching to fallback protocol http2 connIndex=0
2022-06-29T09:21:43Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57243->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:21:43Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57243->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:21:43Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:21:59Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57245->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:21:59Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57245->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:21:59Z INF Retrying connection in up to 4s seconds connIndex=0
2022-06-29T09:22:16Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57251->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:16Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57251->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:16Z INF Retrying connection in up to 8s seconds connIndex=0
2022-06-29T09:22:35Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57254->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:35Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57254->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:35Z INF Retrying connection in up to 16s seconds connIndex=0
2022-06-29T09:23:05Z WRN Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:57258->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:23:05Z WRN Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:57258->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:23:05Z INF Retrying connection in up to 32s seconds connIndex=0
2022-06-29T09:23:32Z WRN Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:23:32Z WRN Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:23:32Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:23:32Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:23:32Z INF Switching to fallback protocol http2 connIndex=0
.
.
.
$ cloudflared tunnel list
ID                                   NAME                      CREATED              CONNECTIONS
4fee3b2c-b661-4e9a-a29f-9399474edfb0 cloudflared-tunnel-test   2022-06-25T08:41:23Z 1xLHR

2022-06-29T09:25:00Z (approx) - switched back to using “Gateway with DoH”

2022-06-29T09:25:26Z INF Connection 8fa82e54-4153-4bec-b1b6-921b5e93f7ed registered connIndex=2 ip=10.10.0.117 location=AMS
2022-06-29T09:25:27Z INF Connection 276d908a-bb49-493a-86ff-e7f1b6135500 registered connIndex=0 ip=198.41.200.233 location=AMS
2022-06-29T09:26:48Z INF Connection 19df8fdb-7148-4bf7-93c7-88c214691399 registered connIndex=3 ip=198.41.192.47 location=LHR
2022-06-29T09:27:26Z INF Lost connection with the edge connIndex=1
2022-06-29T09:27:26Z WRN Serve tunnel error error="connection with edge closed" connIndex=1
2022-06-29T09:27:26Z INF Retrying connection in up to 1s seconds connIndex=1
2022-06-29T09:27:26Z INF Unregistered tunnel connection connIndex=1
2022-06-29T09:27:27Z INF Changing protocol to quic connIndex=1
2022-06-29T09:27:28Z INF Connection d9bc82fb-2276-4384-90a0-8fbe412daf46 registered connIndex=1 ip=198.41.192.167 location=LHR

Hope that all helps a little more.

I noticed a machine that booted during the “Gateway with WARP” period was unable to establish a connection at all. For example, from a boot at approx. 2022-06-29T09:18:00Z…

2022-06-29T09:18:53Z INF Starting tunnel tunnelID=df9d8011-3b48-4aa1-a8de-f480da6607b8
2022-06-29T09:18:53Z INF Version 2022.6.3
2022-06-29T09:18:53Z INF GOOS: darwin, GOVersion: go1.18.3, GoArch: amd64
2022-06-29T09:18:53Z INF Settings: map[token:*****]
2022-06-29T09:18:53Z INF cloudflared will not automatically update if installed by a package manager.
2022-06-29T09:18:53Z INF Generated Connector ID: 642616a5-c34b-427d-9582-047481c00a4c
2022-06-29T09:18:53Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
2022-06-29T09:18:53Z ERR update check failed error="Get \"https://update.argotunnel.com?arch=amd64&clientVersion=2022.6.3&os=darwin\": dial tcp: lookup update.argotunnel.com: no such host"
2022-06-29T09:18:53Z INF Initial protocol quic
2022-06-29T09:18:53Z INF Starting metrics server on 127.0.0.1:49152/metrics
2022-06-29T09:18:53Z ERR Error looking up Cloudflare edge IPs: the DNS query failed error="lookup _origintunneld._tcp.argotunnel.com on [::1]:53: read udp [::1]:53869->[::1]:53: read: connection refused"
2022-06-29T09:18:53Z ERR Please try the following things to diagnose this issue:
2022-06-29T09:18:53Z ERR 1. ensure that argotunnel.com is returning "origintunneld" service records.
2022-06-29T09:18:53Z ERR Run your system's equivalent of: dig srv _origintunneld._tcp.argotunnel.com
2022-06-29T09:18:53Z ERR 2. ensure that your DNS resolver is not returning compressed SRV records.
2022-06-29T09:18:53Z ERR See GitHub issue https://github.com/golang/go/issues/27546
2022-06-29T09:18:53Z ERR For example, you could use Cloudflare's 1.1.1.1 as your resolver:
2022-06-29T09:18:53Z ERR https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/
2022-06-29T09:18:53Z INF Tunnel server stopped
2022-06-29T09:18:53Z ERR Initiating shutdown error="Could not lookup srv records on _origintunneld._tcp.argotunnel.com: lookup _origintunneld._tcp.argotunnel.com on [::1]:53: read udp [::1]:53869->[::1]:53: read: connection refused"
2022-06-29T09:18:54Z INF Metrics server stopped Could not lookup srv records on _origintunneld._tcp.argotunnel.com: lookup _origintunneld._tcp.argotunnel.com on [::1]:53: read udp [::1]:53869->[::1]:53: read: connection refused
2022-06-29T09:18:56Z INF Starting tunnel tunnelID=df9d8011-3b48-4aa1-a8de-f480da6607b8
2022-06-29T09:18:56Z INF Version 2022.6.3
2022-06-29T09:18:56Z INF GOOS: darwin, GOVersion: go1.18.3, GoArch: amd64
2022-06-29T09:18:56Z INF Settings: map[token:*****]
2022-06-29T09:18:56Z INF cloudflared will not automatically update if installed by a package manager.
2022-06-29T09:18:56Z INF Generated Connector ID: 9522c86d-93fa-4621-a0ae-5981d845684e
2022-06-29T09:18:56Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
2022-06-29T09:18:56Z ERR update check failed error="Get \"https://update.argotunnel.com?arch=amd64&clientVersion=2022.6.3&os=darwin\": dial tcp: lookup update.argotunnel.com: no such host"
2022-06-29T09:18:56Z INF Initial protocol quic
2022-06-29T09:18:56Z INF Starting metrics server on 127.0.0.1:49163/metrics
2022-06-29T09:18:56Z ERR Error looking up Cloudflare edge IPs: the DNS query failed error="lookup _origintunneld._tcp.argotunnel.com on [::1]:53: read udp [::1]:61151->[::1]:53: read: connection refused"
2022-06-29T09:18:56Z ERR Please try the following things to diagnose this issue:
2022-06-29T09:18:56Z ERR 1. ensure that argotunnel.com is returning "origintunneld" service records.
2022-06-29T09:18:56Z ERR Run your system's equivalent of: dig srv _origintunneld._tcp.argotunnel.com
2022-06-29T09:18:56Z ERR 2. ensure that your DNS resolver is not returning compressed SRV records.
2022-06-29T09:18:56Z ERR See GitHub issue https://github.com/golang/go/issues/27546
2022-06-29T09:18:56Z ERR For example, you could use Cloudflare's 1.1.1.1 as your resolver:
2022-06-29T09:18:56Z ERR https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/
2022-06-29T09:18:56Z INF Tunnel server stopped
2022-06-29T09:18:56Z ERR Initiating shutdown error="Could not lookup srv records on _origintunneld._tcp.argotunnel.com: lookup _origintunneld._tcp.argotunnel.com on [::1]:53: read udp [::1]:61151->[::1]:53: read: connection refused"
2022-06-29T09:18:57Z INF Metrics server stopped
Could not lookup srv records on _origintunneld._tcp.argotunnel.com: lookup _origintunneld._tcp.argotunnel.com on [::1]:53: read udp [::1]:61151->[::1]:53: read: connection refused
2022-06-29T09:19:02Z INF Starting tunnel tunnelID=df9d8011-3b48-4aa1-a8de-f480da6607b8
2022-06-29T09:19:02Z INF Version 2022.6.3
2022-06-29T09:19:02Z INF GOOS: darwin, GOVersion: go1.18.3, GoArch: amd64
2022-06-29T09:19:02Z INF Settings: map[token:*****]
2022-06-29T09:19:02Z INF cloudflared will not automatically update if installed by a package manager.
2022-06-29T09:19:02Z INF Generated Connector ID: dc65c790-223d-4f09-a21f-6672fb02aee0
2022-06-29T09:19:02Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
2022-06-29T09:19:02Z INF Initial protocol quic
2022-06-29T09:19:02Z INF Starting metrics server on 127.0.0.1:49201/metrics
2022-06-29T09:19:13Z ERR Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:19:13Z ERR Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:19:13Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:19:15Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:19:15Z INF Switching to fallback protocol http2 connIndex=0
2022-06-29T09:19:30Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49250->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:19:30Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49250->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:19:30Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:19:46Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49274->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:19:46Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49274->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:19:46Z INF Retrying connection in up to 4s seconds connIndex=0
2022-06-29T09:20:05Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49327->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:20:05Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49327->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:20:05Z INF Retrying connection in up to 8s seconds connIndex=0
2022-06-29T09:20:25Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49378->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:20:25Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49378->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:20:25Z INF Retrying connection in up to 16s seconds connIndex=0
2022-06-29T09:20:53Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49812->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:20:53Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49812->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:20:53Z INF Retrying connection in up to 32s seconds connIndex=0
2022-06-29T09:21:23Z ERR Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:21:23Z ERR Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:21:23Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:21:24Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:21:24Z INF Switching to fallback protocol http2 connIndex=0
2022-06-29T09:21:39Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49818->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:21:39Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49818->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:21:39Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:21:54Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49822->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:21:54Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49822->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:21:54Z INF Retrying connection in up to 4s seconds connIndex=0
2022-06-29T09:22:12Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49825->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:12Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49825->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:12Z INF Retrying connection in up to 8s seconds connIndex=0
2022-06-29T09:22:29Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49829->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:29Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49829->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:29Z INF Retrying connection in up to 16s seconds connIndex=0
2022-06-29T09:22:49Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49830->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:49Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49830->198.41.200.53:7844: i/o timeout" connIndex=0
2022-06-29T09:22:49Z INF Retrying connection in up to 32s seconds connIndex=0
2022-06-29T09:22:55Z ERR Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:22:55Z ERR Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:22:55Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:22:56Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:22:56Z INF Switching to fallback protocol http2 connIndex=0
2022-06-29T09:23:11Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49835->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:23:11Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49835->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:23:11Z INF Retrying connection in up to 1s seconds connIndex=0
2022-06-29T09:23:26Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49836->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:23:26Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49836->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:23:26Z INF Retrying connection in up to 4s seconds connIndex=0
2022-06-29T09:23:45Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49839->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:23:45Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49839->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:23:45Z INF Retrying connection in up to 8s seconds connIndex=0
2022-06-29T09:24:03Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49840->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:24:03Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49840->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:24:03Z INF Retrying connection in up to 16s seconds connIndex=0
2022-06-29T09:24:29Z ERR Unable to establish connection with Cloudflare edge error="TLS handshake with edge error: read tcp 172.16.0.2:49843->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:24:29Z ERR Serve tunnel error error="TLS handshake with edge error: read tcp 172.16.0.2:49843->198.41.200.193:7844: i/o timeout" connIndex=0
2022-06-29T09:24:29Z INF Retrying connection in up to 32s seconds connIndex=0
2022-06-29T09:25:03Z ERR Failed to create new quic connection error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:25:03Z ERR Serve tunnel error error="failed to dial to edge: timeout: no recent network activity" connIndex=0
2022-06-29T09:25:03Z INF Retrying connection in up to 2s seconds connIndex=0
2022-06-29T09:25:04Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not workunless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0
2022-06-29T09:25:04Z INF Switching to fallback protocol http2 connIndex=0
.
.
.

In one of those machines, can you collect a “mtr” for TCP port 7844 against IP 198.41.200.193 when using Gateway with WARP. After you collect that output, also do “warp-diag” and grab the associated ZIP file. Those should be enough to open a support ticket with Cloudflare and explain the problem.

Thanks, will do.

No response to ticket so far, despite

Status:open
Priority: High

:crying_cat_face:

Still no help from the ticket. Issue still present. For the record, current installs and versions

  • Cloudflare WARP [Version: 2022.7.175.0 (20220714.10)]
  • cloudflared [version 2022.7.1 (built 2022-07-06T12:18:24Z)]

Steps to show connections working when using Cloudflare WARP set to “Gateway with DoH”:

  1. Set Cloudflare WARP to “Gateway with DoH”
  2. Restart machine
  3. cloudflared tunnel list shows machine’s tunnel connected via 4 connections
  4. Logs show all is good.

Steps to reproduce issue when using Cloudflare WARP set to “Gateway with WARP”:

  1. Set Cloudflare WARP to “Gateway with WARP”
  2. Restart machine
  3. cloudflared tunnel list shows machine’s tunnel as NOT connected (zero connections)
  4. Logs show all is NOT good (as seen previously). No connections registered.

I have submitted fresh mtr report and warp-diag to ticket today.

As of Sunday 2022-07-31, I can report that this has suddenly started to work (as expected).

cloudflared via WARP is functional once again.

Strange there was no recognition of the issue anywhere in the status page during this last few weeks though.
Hopefully my reports have aided in diagnosing and fixing the issue.