DNS-over-TLS broken with Knot Resolver since today

Since a couple of hours ago, more precisely 13 January 2020 around 14:30 GMT, all my Knot Resolver instances, running 3.2.1 on Debian, started failing to forward DNS requests to 1.1.1.1 over TLS.

Configuration is like this:

policy.add(policy.all(policy.TLS_FORWARD({
         { '1.1.1.1', hostname='cloudflare-dns.com', ca_file='/etc/ssl/certs/ca-certificates.crt' },
})))

Logs say this:

kresd[8129]: [tls_client] TLS handshake with 1.1.1.1#00853 has completed
kresd[8129]: [tls_client] TLS session has not resumed
kresd[8129]: [gnutls] (5) REC[0x55868f25e580]: Preparing Packet Application Data(23) with length: 41 and min pad: 0
kresd[8129]: [gnutls] (5) REC[0x55868f25e580]: Sent Packet[1] Application Data(23) in epoch 2 and length: 63
kresd[8129]: [gnutls] (3) ASSERT: ../../lib/buffers.c[_gnutls_io_read_buffered]:589
kresd[8129]: [gnutls] (3) ASSERT: ../../lib/record.c[_gnutls_recv_int]:1775
kresd[8129]: [gnutls] (5) REC[0x55868f25e580]: SSL 3.3 Application Data packet received. Epoch 2, length: 445
kresd[8129]: [gnutls] (5) REC[0x55868f25e580]: Expected Packet Application Data(23)
kresd[8129]: [gnutls] (5) REC[0x55868f25e580]: Received Packet Application Data(23) with length: 445
kresd[8129]: [gnutls] (5) REC[0x55868f25e580]: Decrypted Packet[0] Handshake(22) with length: 428
kresd[8129]: [gnutls] (3) ASSERT: ../../lib/buffers.c[get_last_packet]:1168
kresd[8129]: [gnutls] (4) HSK[0x55868f25e580]: NEW SESSION TICKET (4) was received. Length 210[424], frag offset 0, frag length: 210, sequence: 0
kresd[8129]: [gnutls] (3) ASSERT: ../../lib/buffers.c[_gnutls_handshake_io_recv_int]:1429
kresd[8129]: [gnutls] (4) HSK[0x55868f25e580]: parsing session ticket message
kresd[8129]: [gnutls] (3) ASSERT: ../../lib/buffers.c[get_last_packet]:1168
kresd[8129]: [gnutls] (4) HSK[0x55868f25e580]: NEW SESSION TICKET (4) was received. Length 210[210], frag offset 0, frag length: 210, sequence: 0
kresd[8129]: [gnutls] (3) ASSERT: ../../lib/buffers.c[_gnutls_handshake_io_recv_int]:1429
kresd[8129]: [gnutls] (4) HSK[0x55868f25e580]: parsing session ticket message
kresd[8129]: [gnutls] (3) ASSERT: ../../lib/record.c[_gnutls_recv_in_buffers]:1577
kresd[8129]: [gnutls] (3) ASSERT: ../../lib/record.c[_gnutls_recv_int]:1775
kresd[8129]: [io] => connection to '1.1.1.1#00853': error processing TLS data, close

So this started happening on 4 different Knot Resolver instances, with similar configuration, on different ISPs, all at the same time. They can perfectly TLS_FORWARD to other DNS servers though. It looks like Cloudflare changed something which triggers this problem?

Hi Fre,

A recent release re-enabled TLS 1.3 including session resumption support for DoT. It looks like an implementation detail (DNS application data preceded by New Session Ticket in the same TLS record with TLS 1.3) results in Knot Resolver getting upset. This problem has been addressed by knot-resolver 4.2.0 (see https://gitlab.labs.nic.cz/knot/knot-resolver/issues/489 and https://www.knot-resolver.cz/2019-08-05-knot-resolver-4.2.0.html).

As an immediate solution, you can try to upgrade your client to use the newer upstream packages (https://www.knot-resolver.cz/download/). Requesting Debian to cherry-pick the patch is another option.
Disabling session resumption unfortunately does not seem to be a configurable option in Knot Resolver.

As hack, you can indirectly disable session ticket support by patching the priority string in the kresd 3.2.1 binary to use the “PFS” class of ciphers instead of “NORMAL”. Do keep the number of characters the same, there are three spaces before “PFS”:

$ strings /usr/sbin/kresd | grep NORMAL
NORMAL:-VERS-TLS1.0:-VERS-TLS1.1:-VERS-SSL3.0:-ARCFOUR-128:-COMP-ALL:+COMP-NULL
$ sudo sed 's/NORMAL/   PFS/' -i /usr/sbin/kresd

Documentation for PFS is available at https://gnutls.org/manual/html_node/Priority-Strings.html

%NO_TICKETS will prevent the advertizing of the TLS session ticket extension. This is implied by the PFS keyword.

The relevant source code that is being patched can be found here: https://gitlab.labs.nic.cz/knot/knot-resolver/blob/v3.2.1/daemon%2Ftls.c#L67

We’ll look into ways to restore compatibility with broken clients, but some trade off will have to be made.

5 Likes

Thanks. Why didn’t find I discover this bug ticket myself :angry:

1 Like