Cloudflare DNS doesn't return NXDOMAIN properly

ressu · September 3, 2023, 6:05pm

TL;DR: Having a domain that is served from Cloudflare DNS in linux resolver search domains breaks resolver. This seems to be due to cloudflare not returning NXDOMAIN for queries.

This stumped me for a while. I discovered this because my linux resolver stopped resolving names outside of my local network. After a long rabbithole this turns out to be because the resolver gets confused by not receiving NXDOMAIN from the authoritative DNS.

While I discovered this with my own domain home.ressukka.net, it seems that this is reproduceable with any domain hosted on cloudflare.

To demonstrate:

ressu@denial ~> dig cloudflare.com @1.1.1.1 +domain=google.com +ndots=3 +short
104.16.133.229
104.16.132.229
ressu@denial ~> dig cloudflare.com @1.1.1.1 +domain=plex.tv +ndots=3 +short
ressu@denial ~> dig cloudflare.com @1.1.1.1 +domain=cloudflare.com +ndots=3 +short
ressu@denial ~> dig cloudflare.com @1.1.1.1 +domain=microsoft.com +ndots=3 +short
104.16.133.229
104.16.132.229

What is happening here is that dig is simulating the linux resolver and trying to look up cloudflare.com with the given search domain. So effectively it’s looking for cloudflare.com.google.com (in the first example), which naturally doesn’t exist. Since the resolver receives a NXDOMAIN status from the DNS it moves on to search for cloudflare.com which then succeeds. But if we use a domain hosted on Cloudflare DNS, the resolver stops resolving as it instead received a NOERROR response. There is no data, but also there is no reason to move forward either.

This is a new(ish) behaviour on Cloudflare DNS and I’ve first observed this about a month ago when some of my services in my home network stopped working. They are using home.ressukka.net as part of their search domain, which is hosted on cloudflare. Since the domain will now always return NOERROR for any query, all workloads will fail.

To make matters worse, this seems to mostly impact recursive queries. If I manually query the authoritative nameserver the correct response is returned:

ressu@denial ~> dig nosuchdomain.cloudflare.com @1.1.1.1

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> nosuchdomain.cloudflare.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54049
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuchdomain.cloudflare.com.   IN      A

;; AUTHORITY SECTION:
cloudflare.com.         300     IN      SOA     ns3.cloudflare.com. dns.cloudflare.com. 2318959390 10000 2400 604800 300

;; Query time: 36 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 03 19:01:55 IST 2023
;; MSG SIZE  rcvd: 100

ressu@denial ~> dig nosuchdomain.cloudflare.com @ns3.cloudflare.com

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> nosuchdomain.cloudflare.com @ns3.cloudflare.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 21644
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuchdomain.cloudflare.com.   IN      A

;; AUTHORITY SECTION:
cloudflare.com.         300     IN      SOA     ns3.cloudflare.com. dns.cloudflare.com. 2318959390 10000 2400 604800 300

;; Query time: 28 msec
;; SERVER: 162.159.0.33#53(ns3.cloudflare.com) (UDP)
;; WHEN: Sun Sep 03 19:02:05 IST 2023
;; MSG SIZE  rcvd: 100

Note that the response from 1.1.1.1 is NOERROR, while response from ns3.cloudflare.com is NXDOMAIN. Both responses are authoritative answers, so they should be both also responding NXDOMAIN.

Laudian · September 3, 2023, 7:02pm

No.
The query to ns3.cloudflare.com is indeed authoritative, as it answers with the aa (authoritative answer) flag:

The query to 1.1.1.1 however is not authoritative, the aa flag is missing:

The answer seems to be correct to me me, and 8.8.8.8 shows the same behaviour:

dig nosuchdomain.cloudflare.com @8.8.8.8

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> nosuchdomain.cloudflare.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12961
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;nosuchdomain.cloudflare.com.   IN      A

;; AUTHORITY SECTION:
cloudflare.com.         300     IN      SOA     ns3.cloudflare.com. dns.cloudflare.com. 2318959390 10000 2400 604800 300

;; Query time: 28 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Sun Sep 03 21:01:39 CEST 2023
;; MSG SIZE  rcvd: 100

ressu · September 3, 2023, 7:11pm

Oh, you’re correct. Mistake on that front.

Yes, Google DNS shows the same behaviour. My point wasn’t that 1.1.1.1 was misbehaving. If you try the same queries with something like nosuchdomain.google.com or nosuchdomain.microsoft.com you’ll notice that both will return NXDOMAIN.

ressu@denial ~> dig nosuchdomain.cloudflare.com @1.1.1.1

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> nosuchdomain.cloudflare.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32354
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuchdomain.cloudflare.com.   IN      A

;; AUTHORITY SECTION:
cloudflare.com.         300     IN      SOA     ns3.cloudflare.com. dns.cloudflare.com. 2318959390 10000 2400 604800 300

;; Query time: 32 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 03 20:08:16 IST 2023
;; MSG SIZE  rcvd: 100

ressu@denial ~> dig nosuchdomain.google.com @1.1.1.1

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> nosuchdomain.google.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 15884
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuchdomain.google.com.       IN      A

;; AUTHORITY SECTION:
google.com.             60      IN      SOA     ns1.google.com. dns-admin.google.com. 560657216 900 900 1800 60

;; Query time: 24 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 03 20:08:21 IST 2023
;; MSG SIZE  rcvd: 102

ressu@denial ~> dig nosuchdomain.microsoft.com @1.1.1.1

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> nosuchdomain.microsoft.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 10259
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuchdomain.microsoft.com.    IN      A

;; AUTHORITY SECTION:
microsoft.com.          300     IN      SOA     ns1-39.azure-dns.com. azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300

;; Query time: 44 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 03 20:08:27 IST 2023
;; MSG SIZE  rcvd: 128

On the other hand domains hosted on Cloudflare DNS will not return NXDOMAIN:

ressu@denial ~> dig nosuchdomain.plex.tv @1.1.1.1

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> nosuchdomain.plex.tv @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21336
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuchdomain.plex.tv.          IN      A

;; AUTHORITY SECTION:
plex.tv.                1800    IN      SOA     jeremy.ns.cloudflare.com. dns.cloudflare.com. 2319243754 10000 2400 604800 1800

;; Query time: 36 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 03 20:10:09 IST 2023
;; MSG SIZE  rcvd: 113

ressu@denial ~> dig nosuchdomain.ressukka.net @1.1.1.1

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> nosuchdomain.ressukka.net @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14780
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuchdomain.ressukka.net.     IN      A

;; AUTHORITY SECTION:
ressukka.net.           1800    IN      SOA     dale.ns.cloudflare.com. dns.cloudflare.com. 2319246603 10000 2400 604800 1800

;; Query time: 32 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 03 20:10:16 IST 2023
;; MSG SIZE  rcvd: 116

ressu@denial ~> dig nosuchdomain.cloudflare.com @1.1.1.1

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> nosuchdomain.cloudflare.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25037
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuchdomain.cloudflare.com.   IN      A

;; AUTHORITY SECTION:
cloudflare.com.         300     IN      SOA     ns3.cloudflare.com. dns.cloudflare.com. 2318959390 10000 2400 604800 300

;; Query time: 36 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 03 20:10:26 IST 2023
;; MSG SIZE  rcvd: 100

This is what is breaking the resolver libraries.

Laudian · September 3, 2023, 9:58pm

Ok, I see what you mean. I think it might be related to DNSSEC validation.
As soon as the DO bit is set, the authoritative nameserver reports NOERROR instead of NXDOMAIN.

The response contains an NSEC record with label nosuch.cloudflare.com, which does not make any sense.

The same happens for other zones, even those with DNSSEC disabled.

 dig nosuch.cloudflare.com +dnssec @ns3.cloudflare.com

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> nosuch.cloudflare.com +dnssec @ns3.cloudflare.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47457
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;nosuch.cloudflare.com.         IN      A

;; AUTHORITY SECTION:
cloudflare.com.         300     IN      SOA     ns3.cloudflare.com. dns.cloudflare.com. 2318959390 10000 2400 604800 300
nosuch.cloudflare.com.  300     IN      NSEC    \000.nosuch.cloudflare.com. RRSIG NSEC TYPE65283
cloudflare.com.         300     IN      RRSIG   SOA 13 2 300 20230904224932 20230902204932 34505 cloudflare.com. DoqOaNcJq724KUQ1os/SjDMCLTnsFAzItpueQDnDRo8JXcH2rKb6SBj6 UQWnI+Ou5Bzeqd5iZj49Q3ZCI3GktQ==
nosuch.cloudflare.com.  300     IN      RRSIG   NSEC 13 3 300 20230904224932 20230902204932 34505 cloudflare.com. +GYtgNFCRgp0EyYFWC1Ur17ey58+bge4cJai+uHVSdcBPCYXB8FinCd5 cF3U8UvHmwrTrSlL9YOlRr40s5DNxw==

;; Query time: 20 msec
;; SERVER: 2400:cb00:2049:1::a29f:21#53(ns3.cloudflare.com) (UDP)
;; WHEN: Sun Sep 03 23:49:32 CEST 2023
;; MSG SIZE  rcvd: 362

dig nosuch.cloudflare.com @ns3.cloudflare.com

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> nosuch.cloudflare.com @ns3.cloudflare.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 37827
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nosuch.cloudflare.com.         IN      A

;; AUTHORITY SECTION:
cloudflare.com.         300     IN      SOA     ns3.cloudflare.com. dns.cloudflare.com. 2318959390 10000 2400 604800 300

;; Query time: 20 msec
;; SERVER: 2400:cb00:2049:1::a29f:7e2#53(ns3.cloudflare.com) (UDP)
;; WHEN: Sun Sep 03 23:49:37 CEST 2023
;; MSG SIZE  rcvd: 94

i40west · September 3, 2023, 10:13pm

See these two pages for the explanation of Cloudflare’s DNSSEC behavior:

ressu · September 3, 2023, 10:26pm

The latter link actually hints to the behaviour we are seeing (in the section " When Cloudflare Lies").

That would seem to confirm that this is indeed a bug in the response. The NSEC is intended, but the fact that NSEC is returned seems to trigger NOERROR response instead of NXDOMAIN. Initially I was worried that this behaviour was by design and the NOERROR was intentional.

Laudian · September 3, 2023, 11:08pm

I’m a bit confused that records are signed even when DNSSEC is disabled for that zone.

I mean, it shouldn’t hurt as long as there is no DS record in the parent zone, but what’s the point of having a menu to enable/disable DNSSEC if the signing happens anyway?

I’m not too sure that this is a bug, as that article is seven years old. It might just be that since there is an NSEC record for every name, the decision was made to respond to every request with NODATA (and thus, NOERROR) instead of NXDOMAIN.

In the end, whatever library you use should probably check if the answer is contained in the answer section instead of relying on the NOERROR rcode to indicate whether the requested record exists.

i40west · September 3, 2023, 11:20pm

DNSSEC should eliminate most NXDOMAIN responses.

It seems like you’re trying to use NXDOMAIN to mean “there is no A record for this name” when what it really means is “this name not only does not exist at all for any record type, but nothing underneath it exists at all, either”.

ressu · September 4, 2023, 9:09am

There are few important points that we need to keep in mind:

This behaviour is about a month old, which means that this was recently introduced.
The tooling I’m using is not my tooling, it’s standard tooling from Linux distributions used in their intended ways

So the article might be 7 years old, but this is newly introduced behaviour and breaks a common use case in resolver libraries.

Let me elaborate a bit on why this all matters. Let’s take this failry standard Linux resolver configuration as an example:

nameserver 1.1.1.1
search home.ressukka.net
options ndots:3

This is something you might see on a Linux machine. The only notable and unpredictable thing you have here is that the domain home.ressukka.net is hosted on Cloudflare DNS. Which shouldn’t be anything special. Since some Linux distributions are switching to things like systemd resolver and others, I’m using the dig and nslookup tools to simulate the same behaviour as raw Linux glibc would do. Another special thing I’m setting in the commands is the ndots parameter, which tells the resolver that instead of only searching for names which have no dots in them, search for domains that have some dots in them. This is standard behaviour in services like Kubernetes.

So with the base information out of the way, let’s simulate this with an example. Let’s assume that I’m trying to reach cloudflare.com from a host with such a configuration:

> nslookup -debug -ndots=3 -domain=home.ressukka.net cloudflare.com 1.1.1.1
Server:         1.1.1.1
Address:        1.1.1.1#53

------------
    QUESTIONS:
        cloudflare.com.home.ressukka.net, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  ressukka.net
        origin = dale.ns.cloudflare.com
        mail addr = dns.cloudflare.com
        serial = 2319284004
        refresh = 10000
        retry = 2400
        expire = 604800
        minimum = 1800
        ttl = 1800
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
*** Can't find cloudflare.com: No answer

That doesn’t seem right, does it? Maybe it’s just nslookup that is broken, let’s try dig:

❯ dig cloudflare.com @1.1.1.1 +domain=home.ressukka.net +ndots=3                               

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> cloudflare.com @1.1.1.1 +domain=home.ressukka.net +ndots=3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39444
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;cloudflare.com.home.ressukka.net. IN   A

;; AUTHORITY SECTION:
ressukka.net.           1800    IN      SOA     dale.ns.cloudflare.com. dns.cloudflare.com. 2319284004 10000 2400 604800 1800

;; Query time: 64 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Mon Sep 04 09:59:23 IST 2023
;; MSG SIZE  rcvd: 123

Both of these tools get confused by the response. And again, these aren’t some obscure tools doing obscure things, these are standard Linux tooling that often act as the reference implementation.

If we use a domain not hosted on Cloudflare DNS, the commands work exactly as expected:

> nslookup -debug -ndots=3 -domain=badwolf.fi cloudflare.com 1.1.1.1
Server:         1.1.1.1
Address:        1.1.1.1#53

------------
    QUESTIONS:
        cloudflare.com.badwolf.fi, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  badwolf.fi
        origin = ns-1975.awsdns-54.co.uk
        mail addr = awsdns-hostmaster.amazon.com
        serial = 1
        refresh = 7200
        retry = 900
        expire = 1209600
        minimum = 86400
        ttl = 889
    ADDITIONAL RECORDS:
------------
** server can't find cloudflare.com.badwolf.fi: NXDOMAIN
Server:         1.1.1.1
Address:        1.1.1.1#53

------------
    QUESTIONS:
        cloudflare.com, type = A, class = IN
    ANSWERS:
    ->  cloudflare.com
        internet address = 104.16.132.229
        ttl = 207
    ->  cloudflare.com
        internet address = 104.16.133.229
        ttl = 207
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name:   cloudflare.com
Address: 104.16.132.229
Name:   cloudflare.com
Address: 104.16.133.229

> dig cloudflare.com @1.1.1.1 +domain=badwolf.fi +ndots=3

; <<>> DiG 9.18.12-1ubuntu1.1-Ubuntu <<>> cloudflare.com @1.1.1.1 +domain=badwolf.fi +ndots=3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53361
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;cloudflare.com.                        IN      A

;; ANSWER SECTION:
cloudflare.com.         220     IN      A       104.16.132.229
cloudflare.com.         220     IN      A       104.16.133.229

;; Query time: 20 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Mon Sep 04 10:02:49 IST 2023
;; MSG SIZE  rcvd: 75

The fix in my case is easy, I can simply drop the Cloudflare DNS hosted domain from search domains, the reason why I’m bringing this up is that this is actually breaking Linux machines in a very unpredictable way.

Laudian · September 4, 2023, 10:07am

But the tools aren’t broken, they just don’t return the result that you were expecting.

Concatenating names for lookups is really only a valid strategy if you control the queried nameservers and are thus able to guarantee that the concatenated names do not exist randomly.

But you’re querying nameservers that you do not control, hoping for a negative response. That’s essentially crossing your fingers, which is not a very reliable strategy as you noticed.

DarkDeviL · September 4, 2023, 1:28pm

To continue what @i40west mentioned above:

In the earlier standards, it wasn’t apparently “clear enough” when and how to use e.g. RCODE 3 (NXDOMAIN), or RCODE 0 (NOERROR) with an empty data set.

That has fortunately been corrected now, with RFC8020.

As mentioned, if you have something ON or BELOW the label home.ressukka.net, home.ressukka.net is supposed to respond with RCODE 0 (NOERROR), even if the current query type (A, AAAA, TXT, …) does not have any record.

Now, to quote @Laudian 's output from above:

Laudian:

 dig nosuch.cloudflare.com +dnssec @ns3.cloudflare.com

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> nosuch.cloudflare.com +dnssec @ns3.cloudflare.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47457

[...]

;; AUTHORITY SECTION:
cloudflare.com.         300     IN      SOA     ns3.cloudflare.com. dns.cloudflare.com. 2318959390 10000 2400 604800 300
nosuch.cloudflare.com.  300     IN      NSEC    \000.nosuch.cloudflare.com. RRSIG NSEC TYPE65283
cloudflare.com.         300     IN      RRSIG   SOA 13 2 300 20230904224932 20230902204932 34505 cloudflare.com. DoqOaNcJq724KUQ1os/SjDMCLTnsFAzItpueQDnDRo8JXcH2rKb6SBj6 UQWnI+Ou5Bzeqd5iZj49Q3ZCI3GktQ==
nosuch.cloudflare.com.  300     IN      RRSIG   NSEC 13 3 300 20230904224932 20230902204932 34505 cloudflare.com. +GYtgNFCRgp0EyYFWC1Ur17ey58+bge4cJai+uHVSdcBPCYXB8FinCd5 cF3U8UvHmwrTrSlL9YOlRr40s5DNxw==

Continuing with nosuch.cloudflare.com as the example:

There is a NSEC record for nosuch.cloudflare.com , as well as a RRSIG for nosuch.cloudflare.com , - so since those two records are returned, you could argue that nosuch.cloudflare.com is somehow existing, … even if it doesn’t?

Some DNSSEC implementations does however return RCODE 3 (NXDOMAIN), but the ones that do, will not have any records beneath nosuch.cloudflare.com:

If you’re looking at the actual NSEC record, you can see that the NSEC record is claiming that \000.nosuch.cloudflare.com is the next record that actually exists.

Since \000.nosuch.cloudflare.com is (obviously) beneath nosuch.cloudflare.com, nosuch.cloudflare.com itself won’t be able to return RCODE 3 (NXDOMAIN) if it has no records, but only RCODE 0 (NOERROR) with an empty data set.

Serena (name server) told me that she isn’t just helping you, but also openconnectivity.org.

The .ORG registry tells me that openconnectivity.org is NOT currently DNSSEC signed.

Here is what I get from querying that domain:

$ dig cms.openconnectivity.org @1.1.1.1

; <<>> DiG 9.16.37-Debian <<>> cms.openconnectivity.org @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15716
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;cms.openconnectivity.org.      IN      A

;; ANSWER SECTION:
cms.openconnectivity.org. 300   IN      A       104.24.67.16
cms.openconnectivity.org. 300   IN      A       104.24.66.16
cms.openconnectivity.org. 300   IN      A       172.67.67.209

;; Query time: 172 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Sep 04 15:17:15 CEST 2023
;; MSG SIZE  rcvd: 101

$ dig blah.cms.openconnectivity.org @1.1.1.1

; <<>> DiG 9.16.37-Debian <<>> blah.cms.openconnectivity.org @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 22152
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;blah.cms.openconnectivity.org. IN      A

;; AUTHORITY SECTION:
openconnectivity.org.   1800    IN      SOA     dave.ns.cloudflare.com. dns.cloudflare.com. 2319317568 10000 2400 604800 1800

;; Query time: 12 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Sep 04 15:17:21 CEST 2023
;; MSG SIZE  rcvd: 120

$ dig blah.cms.openconnectivity.org @1.1.1.1 +dnssec

; <<>> DiG 9.16.37-Debian <<>> blah.cms.openconnectivity.org @1.1.1.1 +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5883
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;blah.cms.openconnectivity.org. IN      A

;; AUTHORITY SECTION:
openconnectivity.org.   1800    IN      SOA     dave.ns.cloudflare.com. dns.cloudflare.com. 2319317568 10000 2400 604800 1800

;; Query time: 12 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Sep 04 15:17:31 CEST 2023
;; MSG SIZE  rcvd: 120

So one possible way, if you really insist on RCODE 3 (NXDOMAIN), could be to disable DNSSEC again.

In order words, to make your choice between:

DNSSEC && RCODE 0 (NOERROR).
No DNSSEC && RCODE 3 (NXDOMAIN).

Any chance the discrepancy … that happened around one month ago, is because you enabled DNSSEC for your domain around that time?

The \000.nosuch.cloudflare.com record, as mentioned above, still seems very consistent with @i40west 's article from above, even though it correctly turned 7 years old a few months back.

ressu · September 4, 2023, 1:31pm

That is correct. Not only is it unreliable, it’s outright disaster for security. But as you notice in my last example, I am using a domain that I control. The domain home.ressukka.net is specifically intended for domain search lookups.

The reason why I’m using cloudflare.com and other domains as an example is to eliminate possible misconfiguration in the domain on my part.

ressu · September 4, 2023, 1:51pm

It’s true that there is content under home.ressukka.net, but there wouldn’t be any content under cloudflare.com.home.ressukka.net or even nonexistent.home.ressukka.net, which again should return NXDOMAIN.

I do find it interesting that if DNSSEC isn’t enabled in a domain, the behaviour is exactly as we expect and NXDOMAIN is returned.

I wish it was something like that, but the dnssec has been on for a long time now. There are some changes to the records in the domain every now and then due to automation, but those are just adding and deleting individual records.

DarkDeviL · September 4, 2023, 2:15pm

No, it shouldn’t:

$ dig cloudflare.com.home.ressukka.net @1.1.1.1 +dnssec

; <<>> DiG 9.16.37-Debian <<>> cloudflare.com.home.ressukka.net @1.1.1.1 +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9662
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;cloudflare.com.home.ressukka.net. IN   A

;; AUTHORITY SECTION:
ressukka.net.           1800    IN      SOA     dale.ns.cloudflare.com. dns.cloudflare.com. 2319284004 10000 2400 604800 1800
ressukka.net.           1800    IN      RRSIG   SOA 13 2 1800 20230905145432 20230903125432 34505 ressukka.net. wzGj3CA3KIY8SHGjxnM3LEWXggSEjYZwakYO9kmnHuvbGFy+ZGqEFGwt gt7iCKUK8w37/bIexRaGCnF9WLK/dQ==
cloudflare.com.home.ressukka.net. 1800 IN NSEC  \000.cloudflare.com.home.ressukka.net. RRSIG NSEC TYPE65283
cloudflare.com.home.ressukka.net. 1800 IN RRSIG NSEC 13 5 1800 20230905145432 20230903125432 34505 ressukka.net. OHEIjyqPvGbbYsiaI960Zt2aSBASpkOe+QvVf+eQcq7xdIvXZALLdHl+ x9IW/hDDJmqEElEG2dCaeffKDWpo+g==

;; Query time: 32 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Sep 04 15:54:32 CEST 2023
;; MSG SIZE  rcvd: 398

$ dig nonexistent.home.ressukka.net @1.1.1.1 +dnssec

; <<>> DiG 9.16.37-Debian <<>> nonexistent.home.ressukka.net @1.1.1.1 +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15235
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;nonexistent.home.ressukka.net. IN      A

;; AUTHORITY SECTION:
ressukka.net.           1800    IN      SOA     dale.ns.cloudflare.com. dns.cloudflare.com. 2319284004 10000 2400 604800 1800
ressukka.net.           1800    IN      RRSIG   SOA 13 2 1800 20230905145437 20230903125437 34505 ressukka.net. sUMGUoSE1JlonSg9RFYobg6wCCkCsbsJyoqV6Li4wWvzbqp498UY0cDj 14Th7fsY++8W3kKoc/6a+CUoftQm2g==
nonexistent.home.ressukka.net. 1800 IN  NSEC    \000.nonexistent.home.ressukka.net. RRSIG NSEC TYPE65283
nonexistent.home.ressukka.net. 1800 IN  RRSIG   NSEC 13 4 1800 20230905145437 20230903125437 34505 ressukka.net. NQhMupRRbxGdn1GbS6uljE4lqjG95xm69VK/dKX1PNWB8rAPt9zRU7g9 /DyrRPcE42EXEntDXQEs+xHp0j0SiA==

;; Query time: 8 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Sep 04 15:54:37 CEST 2023
;; MSG SIZE  rcvd: 392

The two NSEC records in that output:

cloudflare.com.home.ressukka.net. 1800 IN NSEC  \000.cloudflare.com.home.ressukka.net. RRSIG NSEC TYPE65283

and

nonexistent.home.ressukka.net. 1800 IN  NSEC    \000.nonexistent.home.ressukka.net. RRSIG NSEC TYPE65283

says that \000.cloudflare.com.home.ressukka.net and \000.nonexistent.home.ressukka.net does exist.

As such, neither cloudflare.com.home.ressukka.net, com.home.ressukka.net, home.ressukka.net (given #1), nor nonexistent.home.ressukka.net (given #2) are able to respond with RCODE 3 (NXDOMAIN), since that \000 record mentioned is said to be existing beneath them.

Neither Cloudflare, nor any other DNS providers would ever be able to return RCODE 3 (NXDOMAIN), since those \000 (dummy) records are beneath the label you’re querying for.

And for as long as you’ve had that enabled, you’ve had the RCODE 0 (NOERROR) responses.

To me, it sounds like you need to look somewhere else ( … outside Cloudflare ) for the underlying issue.

Maybe somewhere around all that switching?

system · September 19, 2023, 2:15pm

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.