QUIC not always working over IPv6

Website, Application, Performance DNS & Network
1

Hello,

lately I’ve been noticing ~30s delays upon first visit of some websites hosted on CF and I’ve debugged the issue - it’s because CF is not responding to QUIC request over IPv6, but only for some servers and apparently only for some remote networks.

One such website for example is www.alpinashop.si - traceroute from ISP AS34779, traffic going through CIX (Croatian Internet Exchange):

~]# traceroute -6 -I www.alpinashop.si
traceroute to www.alpinashop.si (2606:4700:3030::6815:30d3), 30 hops max, 80 byte packets
 2  2a01-260-1-1--92.core6.t-2.net (2a01:260:1:1::92)  0.707 ms  0.705 ms  0.746 ms
 3  2001:7f8:28::31:0 (2001:7f8:28::31:0)  2.721 ms  2.842 ms  2.840 ms
 4  2606:4700:3030::6815:30d3 (2606:4700:3030::6815:30d3)  3.094 ms  3.143 ms  3.139 ms

Website sends header over HTTP/2, telling me it supports QUIC:

< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

I can easily reproduce the problem by using custom build of curl3 with integrated QUIC support and forcing IPv6 and QUIC protocol:

~]# curl3 -6 --http3 -o /dev/null --verbose https://www.alpinashop.si/
*   Trying 2606:4700:3037::ac43:8977:443...
* Connect socket 5 over QUIC to 2606:4700:3037::ac43:8977:443
* Sent QUIC client Initial, ALPN: h3,h3-29,h3-28,h3-27

I can see my sent packets in tcpdump, but I’m not getting anything back:

16:09:08.002425 IP6 2a01:260::xxx.34310 > 2606:4700:3037::ac43:8977.443: UDP, length 1200
16:09:09.001432 IP6 2a01:260::xxx.34310 > 2606:4700:3037::ac43:8977.443: UDP, length 1200
16:09:10.999436 IP6 2a01:260::xxx.34310 > 2606:4700:3037::ac43:8977.443: UDP, length 1200

This only happens with IPv6 and HTTP/3 (udp).

Oddly enough, this is working for www.cloudflare.com, which is taking the same traceroute path as previous example, which signals this might be internal CF issue and not an ISP issue:

~]# curl3 -6 --http3 -o /dev/null --verbose https://www.cloudflare.com/                                      
*   Trying 2606:4700::6810:7b60:443...
* Connect socket 5 over QUIC to 2606:4700::6810:7b60:443
* Sent QUIC client Initial, ALPN: h3,h3-29,h3-28,h3-27                                                                                                                                                                                        * Connected to www.cloudflare.com () port 443 (#0)     
* h3 [:method: GET]                                                                                                                                                                                                                           * h3 [:path: /]                           
* h3 [:scheme: https]                                                                                                  
* h3 [:authority: www.cloudflare.com]                   
* h3 [user-agent: curl/7.79.1]                                                                                         
* h3 [accept: */*]        
* Using HTTP/3 Stream ID: 0 (easy handle 0x55a20e76e180)
> GET / HTTP/3                                                                                                         
> Host: www.cloudflare.com                            
> user-agent: curl/7.79.1
> accept: */*
> 
< HTTP/3 200
..
< server: cloudflare
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
< 
{ [10695 bytes data]
100  267k    0  267k    0     0  2287k      0 --:--:-- --:--:-- --:--:-- 2304k
* Connection #0 to host www.cloudflare.com left intact

I’ve tested this with two other ISPs and both websites work there, but they’re routed through different internet exchanges (one through BIX.HU and another through DE-CIX).

2

Hi @Fluke sorry to know that you’re having difficulties and thank you for reporting the issue.
In order for us to scope the investigation and assess impact, could you kindly share the following information from the affected device/network?

  • result of visiting https://<Cloudflare_site>/cdn-cgi/trace
  • IPv6 traceroutes (e.g. sudo mtr -6 -zrwc 20 <Cloudflare_site>)
    If you are an existing customer, you may also open a support ticket with us so you can share the client’s information securely.
    You may cross reference this thread’s link in the ticket and the ticket number here so we can follow up accordingly.
3

Hello,

attaching required data… note I have to request data over http/2, beucase I’m not getting any response over http/3:

~]# curl3 -6 --http2 https://www.alpinashop.si/cdn-cgi/trace
fl=124f20
h=www.alpinashop.si
ip=2a01:260:4106:1::666
ts=1642060434.335
visit_scheme=https
uag=curl/7.79.1
colo=ZAG
http=http/2
loc=SI
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off

~]# mtr -6 -zrwnc 20 www.alpinashop.si
Start: 2022-01-13T08:55:32+0100
HOST: thor.krneki.org                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS34779  2a01:260:4106::1            0.0%    20    1.6   2.1   0.5   6.2   1.9
  2. AS34779  2a01:260:1:1::92            0.0%    20    0.7   0.7   0.6   0.9   0.1
  3. AS???    2001:7f8:28::31:0           0.0%    20    2.6   2.6   2.6   2.7   0.0
  4. AS13335  2606:4700:3030::6815:30d3   0.0%    20    3.1   3.1   3.1   3.2   0.0
4

Hi @Fluke , thanks for providing that information! Is there any chance you could run the same mtr but for www.cloudflare.com? Thanks!

5

Sure, www.cloudflare.com actually works over http/3, but path is identical, this is why I suspect the problem is within your network. Some other sites that are also not working: linegee.net, daringfireball.net, articlewedding.com, www.audiosciencereview.com.

~]# curl3 -6 --http3 https://www.cloudflare.com/cdn-cgi/trace
fl=124f2
h=www.cloudflare.com
ip=2a01:260:4106:1::666
ts=1642084298.859
visit_scheme=https
uag=curl/7.79.1
colo=ZAG
http=http/3
loc=SI
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off

~]# mtr -6 -zrwnc 20 www.cloudflare.com
Start: 2022-01-13T15:32:33+0100
HOST: thor.krneki.org               Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS34779  2a01:260:4106::1       0.0%    20    5.7   2.0   0.4   7.3   2.5
  2. AS34779  2a01:260:1:1::92       0.0%    20    0.5   0.7   0.5   0.9   0.1
  3. AS???    2001:7f8:28::31:0      0.0%    20    2.6   2.6   2.6   2.7   0.0
  4. AS13335  2606:4700::6810:7c60   0.0%    20    3.1   3.2   3.1   4.5   0.3
6

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.