Only at cloudflare: EDE: 6 (DNSSEC Bogus): 'failed to verify DS

stephan7 · March 28, 2022, 5:49pm

Hi Cloudflare,

we’re currently working to upgrade our infrastructure to also be DNSSEC capable and while trying different option we have noticed that our test domain resolves by any other common public resolver (e.g. google, quad9, opendns) but not by 1.1.1.1 (Cloudflare).

We first noticed a problem as the domain stopped working using 1.1.1.1 “just regular”:

# kdig igmp.de @1.1.1.1
;; ->>HEADER<<- opcode: QUERY; status: SERVFAIL; id: 17605
;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; igmp.de.            		IN	A

;; Received 25 B
;; Time 2022-03-28 19:31:28 CEST
;; From [email protected](UDP) in 109.9 ms

adding +dnssec for more details shows the titled error:

# kdig igmp.de soa +dnssec @1.1.1.1
;; ->>HEADER<<- opcode: QUERY; status: SERVFAIL; id: 35310
;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: do; UDP size: 1232 B; ext-rcode: NOERROR
;; **EDE: 6 (DNSSEC Bogus): 'failed to verify DS igmp.de.: using DNSKEY ids = [9795]'**

;; QUESTION SECTION:
;; igmp.de.            		IN	SOA

;; Received 97 B
;; Time 2022-03-28 19:31:58 CEST
;; From [email protected](UDP) in 113.7 ms

testing the same against Q9 (same for 8.8.8.8) shows NOERROR:

# kdig igmp.de soa +dnssec @9.9.9.9
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 41945
;; Flags: qr rd ra ad; QUERY: 1; ANSWER: 2; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: do; UDP size: 1232 B; ext-rcode: NOERROR

;; QUESTION SECTION:
;; igmp.de.            		IN	SOA

;; ANSWER SECTION:
igmp.de.            	3600	IN	SOA	ns1.kuehlbox.de. hostmaster.igmp.de. 1648477658 10800 3600 604800 86400
igmp.de.            	3600	IN	RRSIG	SOA 13 2 3600 20220402142738 20220328132738 9200 igmp.de. A6U8HQ3jbWWJRG9zKL+CHFekHQIhCJY4zDPcUOVxhsMDuL0QdHacRkj1/VwAcbhrgi7v5B9Japh1iFKDpomJrQ==

;; Received 199 B
;; Time 2022-03-28 19:37:06 CEST
;; From [email protected](UDP) in 149.5 ms

While we might have done something wrong at our side - what worries us most here is the inconsistency on the tests between different vendors. Failures only seen so far on 1.1.1.1

Also DENIC NAST tool does past without any warnings/notice:

https://www.denic.de/service/tools/nast/

To simplify things, we have (temporary) removed ZSK (256) key to just use 257 key for signing, however didn’t made any difference.

dnssec-analyzer does not report any problem:

dnsviz also reports all records being secure:

Cloudflares EDE code reports DNSKEY ids = [9795] as failure, interestingly that should be still part of DENIC chain (our key currently is id=9200) which appears very odd.

taking a look at full “drill” shows 9795 (and other ids) as trusted:

# drill -DT -p 53 igmp.de SOA
;; Number of trusted keys: 1
;; Domain: .
[T] . 172800 IN DNSKEY 256 3 8 ;{id = 47671 (zsk), size = 2048b}
. 172800 IN DNSKEY 257 3 8 ;{id = 20326 (ksk), size = 2048b}
. 172800 IN DNSKEY 256 3 8 ;{id = 9799 (zsk), size = 2048b}
Checking if signing key is trusted:
New key: .	172800	IN	DNSKEY	256 3 8 AwEAAZym4HCWiTAAl2Mv1izgTyn9sKwgi5eBxpG29bVlefq/r+TGCtmUElvFyBWHRjvf9mBglIlTBRse22dvzNOI+cYrkjD6LOHuxMoc/d4WtXWKdviNmrtWF2GpjmDOI98gLd4BZ0U/lY847mJP9LypFABZcEn3zM3vce4Ee1A3upSlFQ2TFyJSD9HvMnP4XneFexBxV96RpLcy2O+u2W6ChIiDCjlrowPCcU3zXfXxyWy/VKM6TOa8gNf+aKaVkcv/eIh5er8rrsqAi9KT8O5hmhzYLkUOQEXVSRORV0RMt9l3JSwWxT1MebEDvtfBag3uo+mZwWSFlpc9kuzyWBd72Ec= ;{id = 9799 (zsk), size = 2048b}
	Trusted key: .	86400	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b}
	Trusted key: .	172800	IN	DNSKEY	256 3 8 AwEAAak/ZU9wDNQD7XTAGTDkn32UR8I6auRDekbGky+yyWKdUHmwAJv90YHCUTib8aVBgNgbxkeeZGRx3W4+XhMZbfUr5fMwmD3u9P2yzJpbRtjGNM/XZvzGs9HHNymz3Bp851anHZfNy6pJud265/XMKzFlAY8sMJjum0hvx/DuCDELLyhsvdfOD9rHM93UXO0bcAjvI8tjZsGI+Pfp9KdxF9vS/sAzpFXKsldix+e6xv8rRS6WPg2LAooxF+eO5DgFSilYmnyCK4VPJ7ntjD/8m0bs128ZT1eY3oXCbojDv59lLAgrdGSbcVxQF2KHoUHDmkOC5BzG/1xRtW4v/3y4/H8= ;{id = 47671 (zsk), size = 2048b}
	Trusted key: .	172800	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b}
	Trusted key: .	172800	IN	DNSKEY	256 3 8 AwEAAZym4HCWiTAAl2Mv1izgTyn9sKwgi5eBxpG29bVlefq/r+TGCtmUElvFyBWHRjvf9mBglIlTBRse22dvzNOI+cYrkjD6LOHuxMoc/d4WtXWKdviNmrtWF2GpjmDOI98gLd4BZ0U/lY847mJP9LypFABZcEn3zM3vce4Ee1A3upSlFQ2TFyJSD9HvMnP4XneFexBxV96RpLcy2O+u2W6ChIiDCjlrowPCcU3zXfXxyWy/VKM6TOa8gNf+aKaVkcv/eIh5er8rrsqAi9KT8O5hmhzYLkUOQEXVSRORV0RMt9l3JSwWxT1MebEDvtfBag3uo+mZwWSFlpc9kuzyWBd72Ec= ;{id = 9799 (zsk), size = 2048b}
Key is now trusted!
[T] de. 86400 IN DS 26755 8 2 f341357809a5954311ccb82ade114c6c1d724a75c0395137aa3978035425e78d
;; Domain: de.
[T] de. 300 IN DNSKEY 256 3 8 ;{id = 9795 (zsk), size = 1024b}
de. 300 IN DNSKEY 257 3 8 ;{id = 26755 (ksk), size = 2048b}
Checking if signing key is trusted:
New key: de.	300	IN	DNSKEY	256 3 8 AwEAAbob6Rvo4lELF9StW7S0LthOmTUk1mNq1RxzIwgvyreeZRam94Y/btdDz7XlrZrzAPV9gP5rmL38ujU7MmBd6UhNSBjeSDC/ns77J/AAzXHWxKofQBo46UycZztRJM5CKTuc9FAlBNZ2OukZY4WFPlU0WtLZeQ4ooByJVgdlPXEB ;{id = 9795 (zsk), size = 1024b}
	Trusted key: .	86400	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b}
	Trusted key: .	172800	IN	DNSKEY	256 3 8 AwEAAak/ZU9wDNQD7XTAGTDkn32UR8I6auRDekbGky+yyWKdUHmwAJv90YHCUTib8aVBgNgbxkeeZGRx3W4+XhMZbfUr5fMwmD3u9P2yzJpbRtjGNM/XZvzGs9HHNymz3Bp851anHZfNy6pJud265/XMKzFlAY8sMJjum0hvx/DuCDELLyhsvdfOD9rHM93UXO0bcAjvI8tjZsGI+Pfp9KdxF9vS/sAzpFXKsldix+e6xv8rRS6WPg2LAooxF+eO5DgFSilYmnyCK4VPJ7ntjD/8m0bs128ZT1eY3oXCbojDv59lLAgrdGSbcVxQF2KHoUHDmkOC5BzG/1xRtW4v/3y4/H8= ;{id = 47671 (zsk), size = 2048b}
	Trusted key: .	172800	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b}
	Trusted key: .	172800	IN	DNSKEY	256 3 8 AwEAAZym4HCWiTAAl2Mv1izgTyn9sKwgi5eBxpG29bVlefq/r+TGCtmUElvFyBWHRjvf9mBglIlTBRse22dvzNOI+cYrkjD6LOHuxMoc/d4WtXWKdviNmrtWF2GpjmDOI98gLd4BZ0U/lY847mJP9LypFABZcEn3zM3vce4Ee1A3upSlFQ2TFyJSD9HvMnP4XneFexBxV96RpLcy2O+u2W6ChIiDCjlrowPCcU3zXfXxyWy/VKM6TOa8gNf+aKaVkcv/eIh5er8rrsqAi9KT8O5hmhzYLkUOQEXVSRORV0RMt9l3JSwWxT1MebEDvtfBag3uo+mZwWSFlpc9kuzyWBd72Ec= ;{id = 9799 (zsk), size = 2048b}
	Trusted key: de.	300	IN	DNSKEY	256 3 8 AwEAAbob6Rvo4lELF9StW7S0LthOmTUk1mNq1RxzIwgvyreeZRam94Y/btdDz7XlrZrzAPV9gP5rmL38ujU7MmBd6UhNSBjeSDC/ns77J/AAzXHWxKofQBo46UycZztRJM5CKTuc9FAlBNZ2OukZY4WFPlU0WtLZeQ4ooByJVgdlPXEB ;{id = 9795 (zsk), size = 1024b}
Key is now trusted!
	Trusted key: de.	300	IN	DNSKEY	257 3 8 AwEAAbWUSd/QN9Ae543xzdiacY6qbjwtZ21QfmdgxRdm4Z7bjjHWy249uqxCyjjjoS4LDoRDKmj7ElffMKvTWKE1qFKu0p8TUy4wyhX0M+m5FUjvQ3CiZMi+qY7GSHA5B+Zd73cidmnTeb3e8lso6jEsXg05/VZ2AyAqWF6FexEIFxIqiwwLk4UP0BwZ17Ur3q1qx9VSbPMyHgQ9d6nHUN1EEJsTDA2v0vKumsUyp74ZanRZ/bB/6IzpaaZyr5BLF5pSCNdbRNjVmkwYD0993vm79LueyOeibsoHRc16jhALrIJou1PFjdq7YQsYN0KtqRiJtaAfPprDBREpeamPuW/MnW0= ;{id = 26755 (ksk), size = 2048b}
[T] igmp.de. 86400 IN DS 9200 13 2 e211d199943e9ab637d399609718ef203b27e0fa91df352dd7bfc10756c6f976
;; Domain: igmp.de.
[T] igmp.de. 86400 IN DNSKEY 257 3 13 ;{id = 9200 (ksk), size = 256b}
[T] igmp.de.	3600	IN	SOA	ns1.kuehlbox.de. hostmaster.igmp.de. 1648477658 10800 3600 604800 86400
;;[S] self sig OK; [B] bogus; [T] trusted

Would be great if you can help us to understand the failure you (possible) see here. Thanks in advance!

Cheers,
Stephan

milk · March 28, 2022, 8:03pm

Okay this is very strange, as everything on your end seems correct. Other .de domains resolve fine as well with key 9795 on Cloudflare. Perhaps @mvavrusa knows.

stephan7 · March 28, 2022, 8:36pm

we tested another .de domain as well as a .net domain with similar failure (those are customer owned, cannot public disclose them atm) - so currently the authoritative nameservers are likely the trigger - but we haven’t been able to root cause the reason, hence seeking for your assistance

mvavrusa · March 28, 2022, 10:23pm

Hi! I can reproduce the issue with this domain as well. I’m not sure yet why doesn’t it like the DS signatures, it seems okay to me, opened an internal ticket to investigate it. I added a workaround in the meantime so that it at least resolves.

stephan7 · March 28, 2022, 10:35pm

Hi, thanks a lot for taking a look! Will leave for now the nameservers in that state so the team can investigate. Obviously we want to get confident with DNSSEC and the next step would be to eval other software such as knot - but for now I think its important to identify the discrepancy.

mvavrusa · March 28, 2022, 10:36pm

I think the issue is that the nameserver authoritative for the zone provides the DS record for some reason as well, and it doesn’t get deduplicated before checking the DS record set. We’ll fix this with the next release!

(The authoritative nameserver can also “fix” it by not returning DS when it’s not authoritative for it)

% kdig @ns1.kuehlbox.de igmp.de A +dnssec
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 48241
;; Flags: qr aa rd; QUERY: 1; ANSWER: 2; AUTHORITY: 5; ADDITIONAL: 9
...
;; AUTHORITY SECTION:
igmp.de.            	86400	IN	DS	9200 13 2 E211D199943E9AB637D399609718EF203B27E0FA91DF352DD7BFC10756C6F976
igmp.de.            	3600	IN	NS	ns1.kuehlbox.de.
igmp.de.            	3600	IN	NS	ns2.kuehlbox.de.
igmp.de.            	86400	IN	RRSIG	DS 13 2 86400 20220402142738 20220328132738 9200 igmp.de. YcVZnP+iHvejI7EKs9GqYxvJ5+WoWrdc6td8BVyy4MPkF7oFy7PitmXzurAnOvaNwN3MHabqDP5YFrL51zFP6Q==
igmp.de.            	3600	IN	RRSIG	NS 13 2 3600 20220402142738 20220328132738 9200 igmp.de. u3VasRMlbeUY7ymdUUxJblDhQtte1T//vN/GzTAtKy4/znORp5cUun4CLAnGv1ELYxHnp3G5XXk2KiyQSB18HQ==

stephan7 · March 28, 2022, 10:55pm

Awesome! I think you nailed it down. As for a quick test, removed the DS entry and response now looks like:

;; AUTHORITY SECTION:
igmp.de.            	3600	IN	NS	ns1.kuehlbox.de.
igmp.de.            	3600	IN	NS	ns2.kuehlbox.de.
igmp.de.            	3600	IN	RRSIG	NS 13 2 3600 20220402224650 20220328214650 9200 igmp.de. SW/lKxEJ7GvENTvG3iVf8LyAn9Nba/0z//6Qj6X9gCMuokpH4uUkxSWgJXYlvb5CaMp21hkaqibyDQi1tz1lpQ==

More tests to follow.

mvavrusa · March 29, 2022, 5:37pm

That should work. You don’t have to return anything else than the answer section tbh. The optional authority data is generally not useful for the target records (if it’s not an empty answer or referral), and most nameservers don’t include it by default nowadays.

stephan7 · March 29, 2022, 10:06pm

Verified now with other domains and can confirm those DS entries where what was causing the irritation at 1.1.1.1. So thanks a lot for pointing that out and in the other hand glad we could also contribute to make Cloudflare dns server now little better

system · April 1, 2022, 10:06pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.