Failed to create new quic connection

I’ve spent several hours now scouring for info and found several topics that seem related, but also either resolved months (or years) ago or are not resolved.

Issue 1, maybe it’s the cause of everything: Po

  1. My log shows an issue with post-quantum: I’ve never enabled it and have now explicitly disabled it via config and environment and it’s still enabled. Maybe this is my issue.
  2. Quic always fails to make a new connect, citing failed to dial to edge with quic: timeout: no recent network activity.

Furthermore, I’ve spent time debugging with tcpdump, I can see the start up querying my DNS server and correctly resolving (to my best understanding) the argotunnel.com IPs… but then there’s never any activity after that despite the cloudflared log continuing retrying and getting the above error.

tcpdump:

Summary
08:54:00.185393 IP 10.0.0.6.51257 > <DNS>.53: 35710+ [1au] TXT? cfd-features.argotunnel.com. (56)
08:54:00.185903 IP 10.0.0.6.42693 > <DNS>.53: 38307+ [1au] AAAA? update.argotunnel.com. (50)
08:54:00.185946 IP 10.0.0.6.47669 > <DNS>.53: 30666+ [1au] A? update.argotunnel.com. (50)
08:54:00.201006 IP <DNS>.53 > 10.0.0.6.51257: 35710 1/0/1 TXT "{"pq":101}" (79)
08:54:00.201156 IP 10.0.0.6.59271 > <DNS>.53: 54168+ [1au] TXT? protocol-v2.argotunnel.com. (55)
08:54:00.204796 IP <DNS>.53 > 10.0.0.6.47669: 30666 2/0/1 A 104.18.24.129, A 104.18.25.129 (82)
08:54:00.204902 IP <DNS>.53 > 10.0.0.6.42693: 38307 2/0/1 AAAA 2606:4700::6812:1981, AAAA 2606:4700::6812:1881 (106)
08:54:00.211939 IP <DNS>.53 > 10.0.0.6.59271: 54168 1/0/1 TXT "[{"protocol": "http2", "percentage": 100}, {"protocol": "quic", "percentage": 100}]" (151)
08:54:00.220444 IP 10.0.0.6.42383 > <DNS>.53: 58742+ [1au] SRV? _v2-origintunneld._tcp.argotunnel.com. (66)
08:54:00.231209 IP <DNS>.53 > 10.0.0.6.42383: 58742 2/0/1 SRV region1.v2.argotunnel.com.:7844 1 1, SRV region2.v2.argotunnel.com.:7844 2 1 (156)
08:54:00.231314 IP 10.0.0.6.33359 > <DNS>.53: 19213+ [1au] AAAA? region1.v2.argotunnel.com. (54)
08:54:00.231352 IP 10.0.0.6.43185 > <DNS>.53: 10398+ [1au] A? region1.v2.argotunnel.com. (54)
08:54:00.243188 IP <DNS>.53 > 10.0.0.6.33359: 19213 10/0/1 AAAA 2606:4700:a0::7, AAAA 2606:4700:a0::1, AAAA 2606:4700:a0::6, AAAA 2606:4700:a0::10, AAAA 2606:4700:a0::5, AAAA 2606:4700:a0::8, AAAA 2606:4700:a0::4, AAAA 2606:4700:a0::2, AAAA 2606:4700:a0::9, AAAA 2606:4700:a0::3 (334)
08:54:00.248527 IP <DNS>.53 > 10.0.0.6.43185: 10398 10/0/1 A 198.41.192.37, A 198.41.192.57, A 198.41.192.7, A 198.41.192.67, A 198.41.192.107, A 198.41.192.77, A 198.41.192.27, A 198.41.192.47, A 198.41.192.167, A 198.41.192.227 (214)
08:54:00.248818 IP 10.0.0.6.42076 > <DNS>.53: 38209+ [1au] AAAA? region2.v2.argotunnel.com. (54)
08:54:00.248847 IP 10.0.0.6.55525 > <DNS>.53: 52094+ [1au] A? region2.v2.argotunnel.com. (54)
08:54:00.259314 IP <DNS>.53 > 10.0.0.6.42076: 38209 10/0/1 AAAA 2606:4700:a8::7, AAAA 2606:4700:a8::3, AAAA 2606:4700:a8::2, AAAA 2606:4700:a8::9, AAAA 2606:4700:a8::8, AAAA 2606:4700:a8::10, AAAA 2606:4700:a8::6, AAAA 2606:4700:a8::5, AAAA 2606:4700:a8::4, AAAA 2606:4700:a8::1 (334)
08:54:00.265337 IP <DNS>.53 > 10.0.0.6.55525: 52094 10/0/1 A 198.41.200.53, A 198.41.200.193, A 198.41.200.13, A 198.41.200.113, A 198.41.200.73, A 198.41.200.33, A 198.41.200.233, A 198.41.200.43, A 198.41.200.63, A 198.41.200.23 (214)
08:54:00.269223 IP 10.0.0.6.47968 > <DNS>.53: 27420+ [1au] SRV? _v2-origintunneld._tcp.argotunnel.com. (66)
08:54:00.283038 IP <DNS>.53 > 10.0.0.6.47968: 27420 2/0/1 SRV region1.v2.argotunnel.com.:7844 1 1, SRV region2.v2.argotunnel.com.:7844 2 1 (156)
08:54:00.283139 IP 10.0.0.6.36912 > <DNS>.53: 8115+ [1au] AAAA? region1.v2.argotunnel.com. (54)
08:54:00.283175 IP 10.0.0.6.35936 > <DNS>.53: 62452+ [1au] A? region1.v2.argotunnel.com. (54)
08:54:00.294649 IP <DNS>.53 > 10.0.0.6.36912: 8115 10/0/1 AAAA 2606:4700:a0::7, AAAA 2606:4700:a0::1, AAAA 2606:4700:a0::6, AAAA 2606:4700:a0::10, AAAA 2606:4700:a0::5, AAAA 2606:4700:a0::8, AAAA 2606:4700:a0::4, AAAA 2606:4700:a0::2, AAAA 2606:4700:a0::9, AAAA 2606:4700:a0::3 (334)
08:54:00.294844 IP <DNS>.53 > 10.0.0.6.35936: 62452 10/0/1 A 198.41.192.37, A 198.41.192.57, A 198.41.192.7, A 198.41.192.67, A 198.41.192.107, A 198.41.192.77, A 198.41.192.27, A 198.41.192.47, A 198.41.192.167, A 198.41.192.227 (214)
08:54:00.295127 IP 10.0.0.6.43075 > <DNS>.53: 6529+ [1au] AAAA? region2.v2.argotunnel.com. (54)
08:54:00.295154 IP 10.0.0.6.51150 > <DNS>.53: 56006+ [1au] A? region2.v2.argotunnel.com. (54)
08:54:00.306559 IP <DNS>.53 > 10.0.0.6.43075: 6529 10/0/1 AAAA 2606:4700:a8::7, AAAA 2606:4700:a8::3, AAAA 2606:4700:a8::2, AAAA 2606:4700:a8::9, AAAA 2606:4700:a8::8, AAAA 2606:4700:a8::10, AAAA 2606:4700:a8::6, AAAA 2606:4700:a8::5, AAAA 2606:4700:a

cloudflared:

Summary
2023-12-02T08:54:00Z DBG Loading configuration from /root/.cloudflared/config.yml
2023-12-02T08:54:00Z INF Starting tunnel tunnelID=47905d3a-6edd-4b9b-a65c-fd80678c1bd5
2023-12-02T08:54:00Z INF Version 2023.10.0
2023-12-02T08:54:00Z INF GOOS: linux, GOVersion: go1.20.6, GoArch: amd64
2023-12-02T08:54:00Z INF Settings: map[config:/root/.cloudflared/config.yml cred-file:/root/.cloudflared/47905d3a-6edd-4b9b-a65c-fd80678c1bd5.json credentials-file:/root/.cloudflared/47905d3a-6edd-4b9b-a65c-fd80678c1bd5.json logfile:/root/cloudflared.log loglevel:debug no-autoupdate:true p:quic protocol:quic retries:2]
2023-12-02T08:54:00Z INF Environmental variables map[TUNNEL_POST_QUANTUM:false]
2023-12-02T08:54:00Z INF Generated Connector ID: e7e1f78d-1008-461f-8a44-a599423004b6
2023-12-02T08:54:00Z INF cloudflared will not automatically update if installed by a package manager.
2023-12-02T08:54:00Z DBG Refreshed feature account_hash=32 pq_enabled=true pq_perct=101
2023-12-02T08:54:00Z DBG Fetched protocol: quic
2023-12-02T08:54:00Z INF Initial protocol quic
2023-12-02T08:54:00Z INF ICMP proxy will use 10.0.0.6 as source for IPv4
2023-12-02T08:54:00Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2023-12-02T08:54:00Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 0 is not between ping group 65534 to 65534"
2023-12-02T08:54:00Z DBG ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 0 is not between ping group 65534 to 65534 nor ICMPv6 proxy: socket: permission denied"
2023-12-02T08:54:00Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 0 is not between ping group 65534 to 65534 nor ICMPv6 proxy: socket: permission denied"
2023-12-02T08:54:00Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0
2023-12-02T08:54:00Z DBG edge discovery: resolved edge addresses addresses=["198.41.192.37","198.41.192.57","198.41.192.7","198.41.192.67","198.41.192.107","198.41.192.77","198.41.192.27","198.41.192.47","198.41.192.167","198.41.192.227","2606:4700:a0::7","2606:4700:a0::1","2606:4700:a0::6","2606:4700:a0::10","2606:4700:a0::5","2606:4700:a0::8","2606:4700:a0::4","2606:4700:a0::2","2606:4700:a0::9","2606:4700:a0::3"] event=0
2023-12-02T08:54:00Z DBG edge discovery: resolved edge addresses addresses=["198.41.200.53","198.41.200.193","198.41.200.13","198.41.200.113","198.41.200.73","198.41.200.33","198.41.200.233","198.41.200.43","198.41.200.63","198.41.200.23","2606:4700:a8::7","2606:4700:a8::3","2606:4700:a8::2","2606:4700:a8::9","2606:4700:a8::8","2606:4700:a8::10","2606:4700:a8::6","2606:4700:a8::5","2606:4700:a8::4","2606:4700:a8::1"] event=0
2023-12-02T08:54:00Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0
2023-12-02T08:54:00Z INF Starting metrics server on 127.0.0.1:35705/metrics
2023-12-02T08:54:00Z DBG edge discovery: resolved edge addresses addresses=["198.41.192.37","198.41.192.57","198.41.192.7","198.41.192.67","198.41.192.107","198.41.192.77","198.41.192.27","198.41.192.47","198.41.192.167","198.41.192.227","2606:4700:a0::7","2606:4700:a0::1","2606:4700:a0::6","2606:4700:a0::10","2606:4700:a0::5","2606:4700:a0::8","2606:4700:a0::4","2606:4700:a0::2","2606:4700:a0::9","2606:4700:a0::3"] event=0
2023-12-02T08:54:00Z DBG edge discovery: resolved edge addresses addresses=["198.41.200.53","198.41.200.193","198.41.200.13","198.41.200.113","198.41.200.73","198.41.200.33","198.41.200.233","198.41.200.43","198.41.200.63","198.41.200.23","2606:4700:a8::7","2606:4700:a8::3","2606:4700:a8::2","2606:4700:a8::9","2606:4700:a8::8","2606:4700:a8::10","2606:4700:a8::6","2606:4700:a8::5","2606:4700:a8::4","2606:4700:a8::1"] event=0
2023-12-02T08:54:00Z DBG edge discovery: giving new address to connection connIndex=0 event=0 ip=198.41.200.113
2023-12-02T08:54:05Z INF

===================================================================================
You are hitting an error while using the experimental post-quantum tunnels feature.

Please check:

   https://pqtunnels.cloudflareresearch.com

for known problems.
===================================================================================


2023-12-02T08:54:05Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.113
2023-12-02T08:54:05Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.200.53
2023-12-02T08:54:05Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.200.113
2023-12-02T08:54:06Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.200.53
2023-12-02T08:54:11Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53
2023-12-02T08:54:11Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.57
2023-12-02T08:54:11Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.200.53
2023-12-02T08:54:13Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.57
2023-12-02T08:54:18Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.57
2023-12-02T08:54:18Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.47
2023-12-02T08:54:18Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.57
2023-12-02T08:54:20Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.47
2023-12-02T08:54:25Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.47
2023-12-02T08:54:25Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.227
2023-12-02T08:54:25Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.47
2023-12-02T08:54:25Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.227
2023-12-02T08:54:30Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.227
2023-12-02T08:54:30Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.200.33
2023-12-02T08:54:30Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.227
2023-12-02T08:54:31Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.200.33
2023-12-02T08:54:36Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.33
2023-12-02T08:54:36Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.192.57
2023-12-02T08:54:36Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.200.33
2023-12-02T08:54:38Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.192.57
.........

Everything works fine if I swap to http2 (pq shows as enabled then too, no logged issues). Sniffing the packets again I immediately see activity on port 7844.

Until someone who knows comes along as I’ve not delved into this too deeply, PQ and QUIC has worked fine for us on all our tunnels so not had to debug anything. Just my thoughts…

Do you have any firewalling that might be blocking QUIC UDP packets in or out of your network?
Does your router have any UDP flood protection options enabled that might be getting in the way?

I was concerned about that as well. As far as I can tell, I’ve explicitly allowed outbound udp to the appropriate IPs via iptables, based on this page: https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/deploy-tunnels/tunnel-with-firewall/

iptables -A OUTPUT -p udp -d region1.v2.argotunnel.com --dport 7844
iptables -A OUTPUT -p udp -d region2.v2.argotunnel.com --dport 7844
iptables -A OUTPUT -p udp -d cftunnel.com --dport 7844
iptables -A OUTPUT -p udp -d quic.cftunnel.com --dport 7844

The final two produced DNS errors, as far as I can tell:

iptables v1.8.9 (nf_tables): host/network `cftunnel.com' not found

Externally, this is an LXC in proxmox. I’ve checked through the firewall for both the node and the container and both specify the outbound rule is to accept all. I’ve explicitly enabled the port as well with no difference. I’m not sure what that would look like in my router, but I’ll look.

I can’t find anything about a UDP flood setting on google or going through my router settings.

Try just allowing all traffic out with a destination port 7844. I can’t recall how well iptables works with FQDNs if there are multiple IPs or they change (historically it didn’t work, so I still never do it, but may have changed).

The result of iptables -L:

Summary
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
           udp  --  anywhere             198.41.192.227       udp dpt:7844
           udp  --  anywhere             198.41.192.37        udp dpt:7844
           udp  --  anywhere             198.41.192.107       udp dpt:7844
           udp  --  anywhere             198.41.192.67        udp dpt:7844
           udp  --  anywhere             198.41.192.167       udp dpt:7844
           udp  --  anywhere             198.41.192.57        udp dpt:7844
           udp  --  anywhere             198.41.192.77        udp dpt:7844
           udp  --  anywhere             198.41.192.27        udp dpt:7844
           udp  --  anywhere             198.41.192.7         udp dpt:7844
           udp  --  anywhere             198.41.192.47        udp dpt:7844
           udp  --  anywhere             198.41.200.13        udp dpt:7844
           udp  --  anywhere             198.41.200.113       udp dpt:7844
           udp  --  anywhere             198.41.200.233       udp dpt:7844
           udp  --  anywhere             198.41.200.193       udp dpt:7844
           udp  --  anywhere             198.41.200.23        udp dpt:7844
           udp  --  anywhere             198.41.200.53        udp dpt:7844
           udp  --  anywhere             198.41.200.73        udp dpt:7844
           udp  --  anywhere             198.41.200.63        udp dpt:7844
           udp  --  anywhere             198.41.200.43        udp dpt:7844
           udp  --  anywhere             198.41.200.33        udp dpt:7844
           udp  --  anywhere             anywhere             udp dpt:7844

The final line is the result of iptables -A OUTPUT -p udp --dport 7844, the other lines are the result of the previous iptables commands that I ran. Still no difference.

I tried disabling my routers firewall entirely for a brief moment and found that there was no difference either.

Still hoping someone has a direction to point me in. I don’t know what benefits quic has over http2, but at this point it might just end up where I’m staying for now. Hopefully this is something that’s fixed in future updates otherwise.

With the wealth of information you have it probably is worth reporting to the engineering team here:

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.