US CDC website DNSKEY failures

Hi Cloudflare,

I’ve been noticing sporadic DNSSEC verification failures with the cdc.gov domain when I’m using 1.1.1.1 as my DNS resolver.

I can’t reproduce this scenario when using CDC.gov’s authoritative nameserver or Google’s public DNS resolver (8.8.8.8) or Quad9’s public DNS resolver (9.9.9.9).
It appears that when using Cloudflare DNS - 1.1.1.1 as my DNS resolver, DNSKEY will occasionally fail to be populated as a response. Subsequently, DNSSEC verification will fail because DNSKEY is not populated.

This is the command I am running: kdig +all +additional +authority @1.1.1.1 cdc.gov DNSKEY
Occasionally, the response is empty.
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 59233
;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; cdc.gov. IN DNSKEY

;; Received 25 B
;; Time 2022-04-28 13:22:51 EDT
;; From [email protected](UDP) in 7.7 ms

Occasionally, the response is fulfilled with a proper DNS key.
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 39835
;; Flags: qr rd ra; QUERY: 1; ANSWER: 2; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; cdc.gov. IN DNSKEY

;; ANSWER SECTION:
cdc.gov. 49 IN DNSKEY 257 3 7 AwEAAZQPCbyMlLJZ92jSnb4PpC8AnC8E7PGsOt5z0/IcnkzGPbHS0mVxAD5Et4UcUbXMvQRVEbHLSyPc4WAfs8M2wuV29FZvaf/lkfNBledAsWFet1gAB4K/WChQF5/5IQPjVExnunFEXtG6al5vQqAmHyMfDaFpLzou+Ko1JezXsC/ZPSJz+0q4KrBOrkqLxlINUsgaObWd+KyBgJckechhEMJqjC8KUHD3xps0AZUilNAseF8Mmp6AWPwQzK0CZ3gK82uKlKvPPa5VE153N0ZjWeHY7tyMEUznrTyHzQamjIxnUDVCE0Y0ZnpNFXRgqjfoClHnTw50rXsOGb49RcY1GDM=
cdc.gov. 49 IN DNSKEY 256 3 7 AwEAAc12zuYTsore5udNdOsLKYgTWiPvdszYh6y6ZUgMuuEOzGlNOl0XK9COVQrk7NMFNNXWmQfS3VqrH/PPbi6l5EE90b4UBitfNRpTZBAtlEeyCBe2+qGUCz//5LMEPjWyAIi18v8/SQN2RzxvJgahx75GQyCUljl1aKQ9dTUjZeXH

;; Received 449 B
;; Time 2022-04-28 13:22:51 EDT
;; From [email protected](UDP) in 5.3 ms

Any help/insight your team can provide is much appreciated.

I see sometimes there is no DNSKEY response by Cloudflare. All nameservers of cdc.gov seem correctly supply it.

dig DNSKEY @1.1.1.1 cdc.gov

; <<>> DiG 9.16.1-Ubuntu <<>> DNSKEY @1.1.1.1 cdc.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38338
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;cdc.gov.			IN	DNSKEY

;; Query time: 16 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Apr 28 20:06:47 CEST 2022
;; MSG SIZE  rcvd: 36
dig DNSKEY @1.1.1.1 cdc.gov

; <<>> DiG 9.16.1-Ubuntu <<>> DNSKEY @1.1.1.1 cdc.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12125
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;cdc.gov.			IN	DNSKEY

;; ANSWER SECTION:
cdc.gov.		1051	IN	DNSKEY	257 3 7 AwEAAZQPCbyMlLJZ92jSnb4PpC8AnC8E7PGsOt5z0/IcnkzGPbHS0mVx AD5Et4UcUbXMvQRVEbHLSyPc4WAfs8M2wuV29FZvaf/lkfNBledAsWFe t1gAB4K/WChQF5/5IQPjVExnunFEXtG6al5vQqAmHyMfDaFpLzou+Ko1 JezXsC/ZPSJz+0q4KrBOrkqLxlINUsgaObWd+KyBgJckechhEMJqjC8K UHD3xps0AZUilNAseF8Mmp6AWPwQzK0CZ3gK82uKlKvPPa5VE153N0Zj WeHY7tyMEUznrTyHzQamjIxnUDVCE0Y0ZnpNFXRgqjfoClHnTw50rXsO Gb49RcY1GDM=
cdc.gov.		1051	IN	DNSKEY	256 3 7 AwEAAc12zuYTsore5udNdOsLKYgTWiPvdszYh6y6ZUgMuuEOzGlNOl0X K9COVQrk7NMFNNXWmQfS3VqrH/PPbi6l5EE90b4UBitfNRpTZBAtlEey CBe2+qGUCz//5LMEPjWyAIi18v8/SQN2RzxvJgahx75GQyCUljl1aKQ9 dTUjZeXH

;; Query time: 20 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Apr 28 20:06:47 CEST 2022
;; MSG SIZE  rcvd: 460
1 Like

I can confirm I see it as well. We have existing overrides for this domain and there seems to be some upstreams that don’t resolve it correctly. Let me remove those and see if it improves.

Seems to work fine again now!

I’ll try to remove some of the overrides, but this is still a problem 1.1.1.1 SERVFAIL for multiple US CDC websites - #2 by mvavrusa

It seems to be working for me too. Thanks @mvavrusa.

I’m confused about the other problem on the other thread.
I see that covid.cdc.gov and gis.cdc.gov are both alias’ to Akamai and that when querying the authoritative nameserver ns1.cdc.gov for both DS and DNSKEY; DS returns but no entry for DNSKEY.

Is this still a problem? It seems to be working for me based on dig, kdig, and delv results.

dig @1.1.1.1 covid.cdc.gov A +dnssec +multiline
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11013
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;gis.cdc.gov. IN A

;; ANSWER SECTION:
gis.cdc.gov. 3600 IN CNAME gis.akam.cdc.gov.
gis.cdc.gov. 3600 IN RRSIG CNAME 7 3 3600 (
20220501130129 20220421122309 6677 cdc.gov.
rWJidzo7rKRDIHlNZ5WA+D007rCyX8aQ0r+oSzjOEeBS
or7Ijz++s+coLj6EMqi5DRYfBVtTZHwURfviaMny5jV5
G/bRCB1AU5ZURTutiPRrnddlUJWa/J0j53iA8dxcE0r+
H/7szwBDpMQRcxMJBKE1ikcZwjuAvMDsHwJj8OE= )
gis.akam.cdc.gov. 20 IN A 23.3.115.231
gis.akam.cdc.gov. 20 IN RRSIG A 10 4 20 (
20220501195951 20220428185951 24315 akam.cdc.gov.
ZJoDvob0CHAvm0TT99kcNztc4aVbXfEs8WE87w7VBU5F
HyBGvwsBWzgaGY+XRbHOFrdXYrMq30NVDI06sYGczOh5
raNai+JHAYcL7KUHO5++Co5JPgCI5gDMhKDR9o6jYz2U

delv gis.cdc.gov @1.1.1.1
;; resolution failed: timed out
; fully validated
gis.cdc.gov. 3600 IN CNAME gis.akam.cdc.gov.
gis.cdc.gov. 3600 IN RRSIG CNAME 7 3 3600 20220501130129 20220421122309 6677 cdc.gov. rWJidzo7rKRDIHlNZ5WA+D007rCyX8aQ0r+oSzjOEeBSor7Ijz++s+co Lj6EMqi5DRYfBVtTZHwURfviaMny5jV5G/bRCB1AU5ZURTutiPRrnddl UJWa/J0j53iA8dxcE0r+H/7szwBDpMQRcxMJBKE1ikcZwjuAvMDsHwJj 8OE=

Cloudflare seems to still timeout when checking for DNSKEY for the gis.akam.cdc.gov record, but also indicate that this is fully validated. Other resolvers such as Google DNS (8.8.8.8) and Quad9 (9.9.9.9) seems to be resolving this faster and both of those are fully validated as well.

Hi, the issue there is that cdc.gov delegation has this set of nameservers that are authoritative for it:

% kdig @ns1.cdc.gov cdc.gov NS
...
;; ANSWER SECTION:
cdc.gov.            	86400	IN	NS	icdc-us-ns3.cdc.gov.
cdc.gov.            	86400	IN	NS	ns2.cdc.gov.
cdc.gov.            	86400	IN	NS	ns1.cdc.gov.
cdc.gov.            	86400	IN	NS	ns3.cdc.gov.
cdc.gov.            	86400	IN	NS	icdc-us-ns1.cdc.gov.
cdc.gov.            	86400	IN	NS	icdc-us-ns2.cdc.gov.

All these work for most of the things in cdc.gov, but none of those responds to “akam.cdc.gov”. The only nameservers that do respond for it are the ones from the registrar set (.gov):

% kdig @a.gov-servers.net cdc.gov
...
cdc.gov.            	86400	IN	NS	auth00.ns.uu.net. <-- this works
cdc.gov.            	86400	IN	NS	ns1.cdc.gov.
cdc.gov.            	86400	IN	NS	auth100.ns.uu.net. <-- this works
cdc.gov.            	86400	IN	NS	ns3.cdc.gov.
cdc.gov.            	86400	IN	NS	ns2.cdc.gov.

So there’s 6 nameservers at the child that resolver tries, none of them works. And 5 nameservers at the parent where only 2 of those work. This won’t work correctly until the zone owner for cdc.gov fixes this, or we (or some other resolver) pins the akam.cdc.gov zone to the only two working nameservers, which is not without issues either.

As a workaround we pin nameservers for some subzones, and where it doesn’t work because of reachability, we relay the lookups to other places (which is what caused the DNSKEY lookup failure). I’ll walk the zone sometimes this week to see what other names do we have to pin since the last time, because I haven’t been able to reach anybody at CDC to fix this unfortunately.

1 Like