QUIC CRYPTO_ERROR 0x178 (remote): tls: no application protocol

I’m having trouble getting cloudflared to connect with QUIC, cloudflared falls back to http2 but DNS lookups over UDP are not working and from reading online I’ve seen that QUIC should be used.

This is in an environment that is behind a FortiGate firewall which is not managed by me. I suspect that the firewall might have something to do with this not working because I have deployed cloudflared in other environments and QUIC is used. However the company managing the firewall assure me that they cannot see anything blocked and QUIC over UDP/7488 is allowed.

I’m looking for tips on where to look next, right now the error messages I’m getting don’t point me towards a clear root cause and I don’t know what exactly I should ask the firewall team to check (assuming that the firewall is the root cause, which I strongly suspect).

My full logs are:

Jan 11 12:19:43 localhost systemd[1]: Starting cloudflared...
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF Starting tunnel tunnelID=9f39ac42-cc30-4577-a1e2-45d58926f698
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF Version 2024.1.1
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF GOOS: linux, GOVersion: go1.21.5, GoArch: amd64
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF Settings: map[loglevel:debug no-autoupdate:true token:*****]
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF Generated Connector ID: 537f0267-c556-4038-b85a-747e5f769764
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF cloudflared will not automatically update if installed by a package manager.
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG Fetched protocol: quic
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF Initial protocol quic
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF ICMP proxy will use 10.0.5.63 as source for IPv4
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF ICMP proxy will use fe80::250:56ff:feb7:4271 in zone ens192 as source for IPv6
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG edge discovery: resolved edge addresses addresses=["198.41.192.27","198.41.192.47","198.41.192.167","198.41.192.37","198.41.192.7","198.41.192.227","198.41.192.57","198.41.192.77","198.41.192.67","198.41.192.107","2606:4700:a0::2","2606:4700:a0::9","2606:4700:a0::4","2606:4700:a0::10","2606:4700:a0::5","2606:4700:a0::6","2606:4700:a0::8","2606:4700:a0::3","2606:4700:a0::7","2606:4700:a0::1"] event=0
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG edge discovery: resolved edge addresses addresses=["198.41.200.13","198.41.200.53","198.41.200.193","198.41.200.63","198.41.200.33","198.41.200.23","198.41.200.113","198.41.200.233","198.41.200.43","198.41.200.73","2606:4700:a8::2","2606:4700:a8::8","2606:4700:a8::5","2606:4700:a8::1","2606:4700:a8::3","2606:4700:a8::4","2606:4700:a8::7","2606:4700:a8::9","2606:4700:a8::10","2606:4700:a8::6"] event=0
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF Starting metrics server on 127.0.0.1:32905/metrics
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG edge discovery: resolved edge addresses addresses=["198.41.192.77","198.41.192.57","198.41.192.47","198.41.192.37","198.41.192.7","198.41.192.67","198.41.192.167","198.41.192.27","198.41.192.107","198.41.192.227","2606:4700:a0::3","2606:4700:a0::10","2606:4700:a0::4","2606:4700:a0::7","2606:4700:a0::2","2606:4700:a0::8","2606:4700:a0::1","2606:4700:a0::9","2606:4700:a0::6","2606:4700:a0::5"] event=0
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG edge discovery: resolved edge addresses addresses=["198.41.200.43","198.41.200.233","198.41.200.193","198.41.200.73","198.41.200.113","198.41.200.63","198.41.200.13","198.41.200.53","198.41.200.33","198.41.200.23","2606:4700:a8::2","2606:4700:a8::1","2606:4700:a8::6","2606:4700:a8::5","2606:4700:a8::4","2606:4700:a8::8","2606:4700:a8::7","2606:4700:a8::9","2606:4700:a8::3","2606:4700:a8::10"] event=0
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG edge discovery: giving new address to connection connIndex=0 event=0 ip=198.41.200.113
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.200.113
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:19:43 localhost cloudflared[1648]: 2024-01-11T11:19:43Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.200.113
Jan 11 12:19:45 localhost cloudflared[1648]: 2024-01-11T11:19:45Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:19:45 localhost cloudflared[1648]: 2024-01-11T11:19:45Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:19:45 localhost cloudflared[1648]: 2024-01-11T11:19:45Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.57
Jan 11 12:19:45 localhost cloudflared[1648]: 2024-01-11T11:19:45Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:19:48 localhost cloudflared[1648]: 2024-01-11T11:19:48Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.57
Jan 11 12:19:49 localhost cloudflared[1648]: 2024-01-11T11:19:49Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.192.57
Jan 11 12:19:49 localhost cloudflared[1648]: 2024-01-11T11:19:49Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:19:49 localhost cloudflared[1648]: 2024-01-11T11:19:49Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.57
Jan 11 12:19:52 localhost cloudflared[1648]: 2024-01-11T11:19:52Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:19:52 localhost cloudflared[1648]: 2024-01-11T11:19:52Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:19:52 localhost cloudflared[1648]: 2024-01-11T11:19:52Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.200.233
Jan 11 12:19:52 localhost cloudflared[1648]: 2024-01-11T11:19:52Z INF Retrying connection in up to 16s connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:20:05 localhost cloudflared[1648]: 2024-01-11T11:20:05Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.200.233
Jan 11 12:20:05 localhost cloudflared[1648]: 2024-01-11T11:20:05Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.200.233
Jan 11 12:20:05 localhost cloudflared[1648]: 2024-01-11T11:20:05Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.167
Jan 11 12:20:05 localhost cloudflared[1648]: 2024-01-11T11:20:05Z INF Retrying connection in up to 32s connIndex=0 event=0 ip=198.41.200.233
Jan 11 12:20:35 localhost cloudflared[1648]: 2024-01-11T11:20:35Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.167
Jan 11 12:20:35 localhost cloudflared[1648]: 2024-01-11T11:20:35Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.192.167
Jan 11 12:20:35 localhost cloudflared[1648]: 2024-01-11T11:20:35Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.200.193
Jan 11 12:20:35 localhost cloudflared[1648]: 2024-01-11T11:20:35Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.167
Jan 11 12:21:02 localhost cloudflared[1648]: 2024-01-11T11:21:02Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.200.193
Jan 11 12:21:02 localhost cloudflared[1648]: 2024-01-11T11:21:02Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.200.193
Jan 11 12:21:02 localhost cloudflared[1648]: 2024-01-11T11:21:02Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.57
Jan 11 12:21:02 localhost cloudflared[1648]: 2024-01-11T11:21:02Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.193
Jan 11 12:21:29 localhost cloudflared[1648]: 2024-01-11T11:21:29Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.57
Jan 11 12:21:29 localhost cloudflared[1648]: 2024-01-11T11:21:29Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.192.57
Jan 11 12:21:29 localhost cloudflared[1648]: 2024-01-11T11:21:29Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:21:29 localhost cloudflared[1648]: 2024-01-11T11:21:29Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.57
Jan 11 12:21:39 localhost cloudflared[1648]: 2024-01-11T11:21:39Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:21:40 localhost cloudflared[1648]: 2024-01-11T11:21:40Z ERR Failed to create new quic connection error="failed to dial to edge with quic: CRYPTO_ERROR 0x178 (remote): tls: no application protocol" connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:21:40 localhost cloudflared[1648]: 2024-01-11T11:21:40Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.200.193
Jan 11 12:21:40 localhost cloudflared[1648]: 2024-01-11T11:21:40Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:21:46 localhost cloudflared[1648]: 2024-01-11T11:21:46Z INF Switching to fallback protocol http2 connIndex=0 event=0 ip=198.41.192.77
Jan 11 12:21:46 localhost cloudflared[1648]: 2024-01-11T11:21:46Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.200.193
Jan 11 12:21:46 localhost cloudflared[1648]: 2024-01-11T11:21:46Z DBG Connecting via http2 connIndex=0 event=0 ip=198.41.200.193
Jan 11 12:21:47 localhost cloudflared[1648]: 2024-01-11T11:21:47Z INF Registered tunnel connection connIndex=0 connection=2af6678a-31a7-4fa5-a869-35b30b655f79 event=0 ip=198.41.200.193 location=vie02 protocol=http2
Jan 11 12:21:47 localhost cloudflared[1648]: 2024-01-11T11:21:47Z DBG edge discovery: giving new address to connection connIndex=1 event=0 ip=198.41.192.57
Jan 11 12:21:47 localhost systemd[1]: Started cloudflared.
Jan 11 12:21:47 localhost cloudflared[1648]: 2024-01-11T11:21:47Z DBG Connecting via http2 connIndex=1 event=0 ip=198.41.192.57
Jan 11 12:21:47 localhost cloudflared[1648]: 2024-01-11T11:21:47Z INF Registered tunnel connection connIndex=1 connection=41fa6418-b744-4782-bb73-2407860fdad6 event=0 ip=198.41.192.57 location=bud01 protocol=http2
Jan 11 12:21:48 localhost cloudflared[1648]: 2024-01-11T11:21:48Z DBG edge discovery: giving new address to connection connIndex=2 event=0 ip=198.41.200.23
Jan 11 12:21:48 localhost cloudflared[1648]: 2024-01-11T11:21:48Z DBG Connecting via http2 connIndex=2 event=0 ip=198.41.200.23
Jan 11 12:21:48 localhost cloudflared[1648]: 2024-01-11T11:21:48Z INF Registered tunnel connection connIndex=2 connection=e2a62239-220b-4adc-a479-c61bf11b49ff event=0 ip=198.41.200.23 location=vie06 protocol=http2
Jan 11 12:21:49 localhost cloudflared[1648]: 2024-01-11T11:21:49Z INF Updated to new configuration config="{\"warp-routing\":{\"enabled\":true}}" version=3
Jan 11 12:21:49 localhost cloudflared[1648]: 2024-01-11T11:21:49Z DBG edge discovery: giving new address to connection connIndex=3 event=0 ip=198.41.192.47
Jan 11 12:21:49 localhost cloudflared[1648]: 2024-01-11T11:21:49Z DBG Connecting via http2 connIndex=3 event=0 ip=198.41.192.47
Jan 11 12:21:49 localhost cloudflared[1648]: 2024-01-11T11:21:49Z INF Registered tunnel connection connIndex=3 connection=cb5cf580-af93-493f-be19-9ffb9026e236 event=0 ip=198.41.192.47 location=bud01 protocol=http2

Resolved, the issue was indeed on the FortiGate firewall, disabling the webfilter for this traffic fixed it. The firewall was trying to inspect and filter some traffic and this was getting in the way.

2 Likes