TLSv1.3 failing (new) on 1.1.1.1/1.0.0.1/2606:4700:4700::1111/2606:4700:4700::1001 port 853 [DNS over TLS]

Problem: super-high rate (~>90%) of failures in connecting to Cloudflare’s DNS resolvers on port 853 (DNS over TLS) with TLSv1.3, starting a few days ago

Expected: TLSv1.3 works as per documentation at https://developers.cloudflare.com/1.1.1.1/dns-over-tls/

Resolution: none. Workaround is to allow TLSv1.2 (failures still there, but rate <10%)

Further information: below is log from stunnel 5.55 on x86_64-pc-linux-gnu platform Compiled/running with OpenSSL 1.1.1c 28 May 2019.
Please notice error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version

Dec  1 22:17:04 router stunnel: LOG7[25]: Service [dns-cloudflare] started
Dec  1 22:17:04 router stunnel: LOG7[25]: Setting local socket options (FD=3)
Dec  1 22:17:04 router stunnel: LOG7[25]: Option TCP_NODELAY set on local socket
Dec  1 22:17:04 router stunnel: LOG5[25]: Service [dns-cloudflare] accepted connection from ::1:38321
Dec  1 22:17:04 router stunnel: LOG6[25]: failover: round-robin, starting at entry #2
Dec  1 22:17:04 router stunnel: LOG6[25]: s_connect: connecting 2606:4700:4700::1001:853
Dec  1 22:17:04 router stunnel: LOG7[25]: s_connect: s_poll_wait 2606:4700:4700::1001:853: waiting 10 seconds
Dec  1 22:17:04 router stunnel: LOG7[25]: FD=6 events=0x2001 revents=0x0
Dec  1 22:17:04 router stunnel: LOG7[25]: FD=12 events=0x2005 revents=0x0
Dec  1 22:17:04 router stunnel: LOG5[25]: s_connect: connected 2606:4700:4700::1001:853
Dec  1 22:17:04 router stunnel: LOG5[25]: Service [dns-cloudflare] connected remote server from {MY_IP_ADDRESS}:35502
Dec  1 22:17:04 router stunnel: LOG7[25]: Setting remote socket options (FD=12)
Dec  1 22:17:04 router stunnel: LOG7[25]: Option TCP_NODELAY set on remote socket
Dec  1 22:17:04 router stunnel: LOG7[25]: Remote descriptor (FD=12) initialized
Dec  1 22:17:04 router stunnel: LOG6[25]: SNI: sending servername: 2606:4700:4700::1111
Dec  1 22:17:04 router stunnel: LOG6[25]: Peer certificate required
Dec  1 22:17:04 router stunnel: LOG7[25]: TLS state (connect): before SSL initialization
Dec  1 22:17:04 router stunnel: LOG7[25]: TLS state (connect): SSLv3/TLS write client hello
Dec  1 22:17:04 router stunnel: LOG7[25]: TLS alert (read): fatal: protocol version
Dec  1 22:17:04 router stunnel: LOG3[25]: SSL_connect: ../ssl/record/rec_layer_s3.c:1535: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version
Dec  1 22:17:04 router stunnel: LOG5[25]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
Dec  1 22:17:04 router stunnel: LOG7[25]: Deallocating application specific data for session connect address
Dec  1 22:17:04 router stunnel: LOG7[25]: Remote descriptor (FD=12) closed
Dec  1 22:17:04 router stunnel: LOG7[25]: Local descriptor (FD=3) closed
Dec  1 22:17:04 router stunnel: LOG7[25]: Service [dns-cloudflare] finished (0 left)

You seem to be using SSL 3 (essentially TLS 1.0), not TLS 1.3.

You seem to be using SSL 3 (essentially TLS 1.0), not TLS 1.3.

Most definitely not … it is TLSv1.3. Did you try to reproduce?

Here’s more detail as to what is going on with the failures.

$ openssl s_client -tls1_3 -connect 1.1.1.1:853 -servername cloudflare-dns.com
CONNECTED(00000003)
140296244294848:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:../ssl/record/rec_layer_s3.c:1535:SSL alert number 70
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 215 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

I’m seeing the exact same behavior with my OpenWRT router using unbound + stubby. I just stumbled upon this thread, and sure enough, allowing TLS 1.2 in stubby gets queries working again. Something seems to have broken with TLS 1.3 recently as I get the same error as above nowadays.

[email protected]:~# openssl version
OpenSSL 1.1.1d  10 Sep 2019
[email protected]:~# openssl s_client -tls1_3 -connect 1.1.1.1:853 -servername cloudflare-dns.com
CONNECTED(00000003)
139754359668040:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1543:SSL alert number 70
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 242 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Looking at a packet capture I see the following response from the server in place of the expected Server Hello.

Client Hello

Transport Layer Security
    TLSv1 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 237
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 233
            Version: TLS 1.2 (0x0303)
            Random: f3e3fb03c25f998724dd04233edbf18b652c3dbcf7f592dc…
                GMT Unix Time: Aug 30, 2099 16:20:19.000000000 EDT
                Random Bytes: c25f998724dd04233edbf18b652c3dbcf7f592dcdf9ec2a6…
            Session ID Length: 32
            Session ID: 5cbae0271d899fca3aac3c14b5cb0f7038bd9d5c6741ac18…
            Cipher Suites Length: 8
            Cipher Suites (4 suites)
                Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
                Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
                Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)
            Extensions Length: 152
            Extension: server_name (len=23)
                Type: server_name (0)
                Length: 23
                Server Name Indication extension
                    Server Name list length: 21
                    Server Name Type: host_name (0)
                    Server Name length: 18
                    Server Name: cloudflare-dns.com
            Extension: ec_point_formats (len=4)
                Type: ec_point_formats (11)
                Length: 4
                EC point formats Length: 3
                Elliptic curves point formats (3)
                    EC point format: uncompressed (0)
                    EC point format: ansiX962_compressed_prime (1)
                    EC point format: ansiX962_compressed_char2 (2)
            Extension: supported_groups (len=12)
                Type: supported_groups (10)
                Length: 12
                Supported Groups List Length: 10
                Supported Groups (5 groups)
                    Supported Group: x25519 (0x001d)
                    Supported Group: secp256r1 (0x0017)
                    Supported Group: x448 (0x001e)
                    Supported Group: secp521r1 (0x0019)
                    Supported Group: secp384r1 (0x0018)
            Extension: session_ticket (len=0)
                Type: session_ticket (35)
                Length: 0
                Data (0 bytes)
            Extension: encrypt_then_mac (len=0)
                Type: encrypt_then_mac (22)
                Length: 0
            Extension: extended_master_secret (len=0)
                Type: extended_master_secret (23)
                Length: 0
            Extension: signature_algorithms (len=30)
                Type: signature_algorithms (13)
                Length: 30
                Signature Hash Algorithms Length: 28
                Signature Hash Algorithms (14 algorithms)
                    Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: ed25519 (0x0807)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (7)
                    Signature Algorithm: ed448 (0x0808)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (8)
                    Signature Algorithm: rsa_pss_pss_sha256 (0x0809)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (9)
                    Signature Algorithm: rsa_pss_pss_sha384 (0x080a)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (10)
                    Signature Algorithm: rsa_pss_pss_sha512 (0x080b)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (11)
                    Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (4)
                    Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (5)
                    Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (6)
                    Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: RSA (1)
            Extension: supported_versions (len=3)
                Type: supported_versions (43)
                Length: 3
                Supported Versions length: 2
                Supported Version: TLS 1.3 (0x0304)
            Extension: psk_key_exchange_modes (len=2)
                Type: psk_key_exchange_modes (45)
                Length: 2
                PSK Key Exchange Modes Length: 1
                PSK Key Exchange Mode: PSK with (EC)DHE key establishment (psk_dhe_ke) (1)
            Extension: key_share (len=38)
                Type: key_share (51)
                Length: 38
                Key Share extension
                    Client Key Share Length: 36
                    Key Share Entry: Group: x25519, Key Exchange length: 32
                        Group: x25519 (29)
                        Key Exchange Length: 32
                        Key Exchange: 4f4b71aa0128d4a6d67c75a5e3499ea015979c5f8ce58002…

Server Response

Transport Layer Security
    TLSv1 Record Layer: Alert (Level: Fatal, Description: Protocol Version)
        Content Type: Alert (21)
        Version: TLS 1.0 (0x0301)
        Length: 2
        Alert Message
            Level: Fatal (2)
            Description: Protocol Version (70)

If I set min TLS version to 1.2 and max TLS version to 1.3, I can see that the Server Hello message negotiates to use TLS 1.2 for most new connections, but sometimes negotiates to 1.3 while the Client Hello mentions support for 1.2 and 1.3 under supported_versions. Could this issue be impacting only specific 1.1.1.1 backend nodes given that anycast is at play here?

According to https://1.1.1.1/help I am hitting the EWR DC.