1.1.1.1 SERVFAIL resolving deptapps.coe.berkeley.edu

Hi, we think there’s an issue with 1.1.1.1 resolving deptapps.coe.berkeley.edu. This is especially affecting our Firefox users who have DNS over HTTPS set using Cloudflare as the provider. Below is the diagnostic info. Is this a fix on Cloudflare’s side or something else?

1.1.1.1 SERVFAIL resolving deptapps.coe.berkeley.edu.
8.8.8.8 and other DNS servers correctly resolve the hostname.

; <<>> DiG 9.10.6 <<>> deptapps.coe.berkeley.edu @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 32428
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 06 ("..")
;; QUESTION SECTION:
;deptapps.coe.berkeley.edu.	IN	A

;; Query time: 32 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Sep 09 13:34:58 PDT 2021
;; MSG SIZE  rcvd: 60
$ dig deptapps.coe.berkeley.edu @8.8.8.8

; <<>> DiG 9.10.6 <<>> deptapps.coe.berkeley.edu @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45677
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;deptapps.coe.berkeley.edu.	IN	A

;; ANSWER SECTION:
deptapps.coe.berkeley.edu. 300	IN	CNAME	acg-prod-01.ist.berkeley.edu.
acg-prod-01.ist.berkeley.edu. 10800 IN	A	128.32.189.121

;; Query time: 46 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Sep 09 13:35:33 PDT 2021
;; MSG SIZE  rcvd: 100
$ dig +short CHAOS TXT id.server @1.1.1.1
"SJC"
$ dig +short CHAOS TXT id.server @1.0.0.1
"SJC"
$ dig @ns3.Cloudflare.com whoami.Cloudflare.com txt +short
"135.180.217.185"
$ dig +tcp @1.1.1.1 id.server CH TXT

; <<>> DiG 9.10.6 <<>> +tcp @1.1.1.1 id.server CH TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49252
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;id.server.			CH	TXT

;; ANSWER SECTION:
id.server.		0	CH	TXT	"SJC"

;; Query time: 5 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Sep 09 13:36:20 PDT 2021
;; MSG SIZE  rcvd: 43

$ dig +tcp @1.0.0.1 id.server CH TXT

; <<>> DiG 9.10.6 <<>> +tcp @1.0.0.1 id.server CH TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21092
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;id.server.			CH	TXT

;; ANSWER SECTION:
id.server.		0	CH	TXT	"SJC"

;; Query time: 5 msec
;; SERVER: 1.0.0.1#53(1.0.0.1)
;; WHEN: Thu Sep 09 13:36:45 PDT 2021
;; MSG SIZE  rcvd: 43

DNSViz shows an error when configured to use 1.1.1.1.
https://dnsviz.net/d/deptapps.coe.berkeley.edu/e/745655/dnssec/
Compared to https://dnsviz.net/d/deptapps.coe.berkeley.edu/dnssec/

$ curl -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=cloudflare.com&type=AAAA'
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":true,"CD":false,"Question":[{"name":"cloudflare.com","type":28}],"Answer":[{"name":"cloudflare.com","type":28,"TTL":90,"data":"2606:4700::6810:85e5"},{"name":"cloudflare.com","type":28,"TTL":90,"data":"2606:4700::6810:84e5"}]}```

Here is the same problem, also with another Berkeley domain: Subdomain.domain.edu not showing in 1.1.1.1 - #2 by milk

Can Cloudflare please respond to this?

Any response from Cloudflare reps on this or any of the other posts about resolution issues?

Hi, sorry about the issues resolving these domains. I’ve created an internal ticket to track this and added a workaround. This seems to be an issue with CNAME target into a signed parent zone (berkeley.edu) from an unsigned zone (coe.berkeley.edu) mixed in the same response. I’ll update the thread when it gets resolved.

1 Like

Thanks mvavrusa for the update.

This is now fixed in the SJC PoP, rest of the PoPs should get it by tomorrow. I’ll remove the workarounds then.

1 Like

The domain seems to work again from AMS and WAW locations, awesome! Could you also check this domain: www.kyb.mpg.de . It still gives a SERVFAIL and is a CNAME as well. Should be the same issue – right?

Yes that seems to be the same issue, I’ve added a workaround for this as well until the release is out, which is unfortunately not earlier than this Wednesday.

2 Likes

Thanks mvavrusa. Just confirming that things still look good after the fix to to PoPs.

And thanks @milk for the support in this thread and others.

1 Like

@mvavrusa I have found another domain with a related error, but different DNS Error code (13). From RFC8914 that would be a Cached Error:
The resolver is returning the SERVFAIL RCODE from its cache. It works fine on Google DNS.

[email protected] ~ % dig extranet-t.mpg.de @8.8.8.8

; <<>> DiG 9.10.6 <<>> extranet-t.mpg.de @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31471
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;extranet-t.mpg.de.		IN	A

;; ANSWER SECTION:
extranet-t.mpg.de.	3600	IN	A	134.76.23.72

;; Query time: 99 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Sep 21 18:58:40 CEST 2021
;; MSG SIZE  rcvd: 62

[email protected] ~ % dig extranet-t.mpg.de @1.1.1.1

; <<>> DiG 9.10.6 <<>> extranet-t.mpg.de @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27451
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 0d ("..")
;; QUESTION SECTION:
;extranet-t.mpg.de.		IN	A

;; Query time: 50 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Sep 21 18:58:46 CEST 2021
;; MSG SIZE  rcvd: 52

This also happened again here:

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 28817
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 0d ("..")
;; QUESTION SECTION:
;ab.mpg.de.			IN	A

;; Query time: 141 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Sep 21 18:55:12 CEST 2021
;; MSG SIZE  rcvd: 44

However, it fixed itself after a while:

[email protected] ~ % dig ab.mpg.de @1.1.1.1 +nsid

; <<>> DiG 9.10.6 <<>> ab.mpg.de @1.1.1.1 +nsid
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22153
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; NSID: 37 33 6d 31 37 33 ("73m173")
;; QUESTION SECTION:
;ab.mpg.de.			IN	A

;; ANSWER SECTION:
ab.mpg.de.		3420	IN	A	134.76.31.205

;; Query time: 35 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Sep 21 19:07:25 CEST 2021
;; MSG SIZE  rcvd: 64

It still repeatedly gives a SERVFAIL with the following domains too: edoc.mpg.de, extranet-intern-t.mpg.de and pure.mpg.de.

Added workarounds for these domains as well and a few others.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.