WebEx not resolving

[darren_ata]

Earlier this morning 1.1.1.1 suddenly stopped resolving webex domains for me. It still resolves most other domains without issue. I had to change my DNS provider to 8.8.8.8 to resolve this issue. See below for DIG results (sorry for the image, new user so was being flagged for too many links):

Any ideas? When I use other sites to check the DNS lookup of webex.com using 1.1.1.1 it resolves. I’m stumped. I was on a webex meeting earlier today right before it stopped resolving and made no changes until after it stopped resolving. After it stopped resolving I changed DNS providers to 8.8.8.8 but prefer cloudflare.

[eag082002]

+1 I am affected by this issue as well. Was working fine earlier this morning, then stopped working around the afternoon.

All devices on my network are affected if my DNS resolver is 1.1.1.1 or 1.0.0.1. I can resolve any domains except WebEx. Google DNS and OpenDNS can resolve WebEx domains without an issue.

The interesting thing is if I route my traffic though my backup cellular WAN link (T-Mobile), I can query 1.1.1.1 for any WebEx domains and it resolves without an issue.

My ISP is AT&T.

Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	google.com
Address: 142.250.68.46
Name:	google.com
Address: 2607:f8b0:4007:80e::200e

[email protected]:~ $ nslookup -port=53 google.com 1.0.0.1
Server:		1.0.0.1
Address:	1.0.0.1#53

Non-authoritative answer:
Name:	google.com
Address: 142.250.68.46
Name:	google.com
Address: 2607:f8b0:4007:80e::200e

[email protected]:~ $ nslookup -port=53 webex.com 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

** server can't find webex.com: SERVFAIL

[email protected]:~ $ nslookup -port=53 webex.com 1.0.0.1
Server:		1.0.0.1
Address:	1.0.0.1#53

** server can't find webex.com: SERVFAIL

[email protected]:~ $ nslookup -port=53 webex.com 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	webex.com
Address: 64.68.121.205

[email protected]:~ $ nslookup -port=53 webex.com 208.67.222.222
Server:		208.67.222.222
Address:	208.67.222.222#53

Non-authoritative answer:
Name:	webex.com
Address: 64.68.121.205

[email protected]:~ $ nslookup -port=53 google.com 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	google.com
Address: 142.250.68.46
Name:	google.com
Address: 2607:f8b0:4007:80e::200e

[email protected]:~ $ nslookup -port=53 google.com 1.0.0.1
Server:		1.0.0.1
Address:	1.0.0.1#53

Non-authoritative answer:
Name:	google.com
Address: 142.250.68.46
Name:	google.com
Address: 2607:f8b0:4007:80e::200e```

[cscharff]

Have problems with 1.1.1.1? *Read Me First*

Here are some tips for troubleshooting if you are having issues using Cloudflare’s Resolver. Please take a moment to review them along with the information that will help us to understand and help diagnose any issues. There are two sections to this guide. The first is for troubleshooting name resolution errors/issues and the second is for unreachability or routing issues. New Diagnostic Tool We have a new diagnostic tool which can help gather some of the information requested below. If you copy…

[eag082002]

1.1.1.1 — The free app that makes your Internet faster.

Prefer not to reveal my IP if possible but can do so in a private ticket.

[email protected]:~ $ nslookup webex.com 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

** server can't find webex.com: SERVFAIL

[email protected]:~ $ nslookup webex.com 1.0.0.1
Server:		1.0.0.1
Address:	1.0.0.1#53

** server can't find webex.com: SERVFAIL

[email protected]:~ $ nslookup webex.com 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	webex.com
Address: 64.68.121.205

[email protected]:~ $ nslookup -class=chaos -type=txt id.server 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
id.server	text = "SAN"

Authoritative answers can be found from:

[email protected]:~ $ nslookup -class=chaos -type=txt id.server 1.0.0.1
Server:		1.0.0.1
Address:	1.0.0.1#53

Non-authoritative answer:
id.server	text = "SAN"

Authoritative answers can be found from:

[email protected]:~ $

[darren_ata]

Hope this helps some. I’d need to open up my firewall to allow my client to query DNS outside of my net to use the diagnostic tool but it shows my Cloudflare Data Center as LAX, if that helps. If you need more I can certainly go through the process and get this client set up to fully utilized the diagnostic tool. Thanks.

[email protected]:~ $ dig webex.com @1.1.1.1

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> webex.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 13173
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 16 ("..")
;; QUESTION SECTION:
;webex.com.                     IN      A

;; Query time: 1963 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Feb 25 08:22:25 PST 2021
;; MSG SIZE  rcvd: 44

[email protected]:~ $ dig webex.com @1.0.0.1

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> webex.com @1.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 28360
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 16 ("..")
;; QUESTION SECTION:
;webex.com.                     IN      A

;; Query time: 1963 msec
;; SERVER: 1.0.0.1#53(1.0.0.1)
;; WHEN: Thu Feb 25 08:22:34 PST 2021
;; MSG SIZE  rcvd: 44

[email protected]:~ $ dig webex.com @8.8.8.8

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> webex.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64999
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;webex.com.                     IN      A

;; ANSWER SECTION:
webex.com.              12      IN      A       64.68.121.205

;; Query time: 9 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Feb 25 08:22:49 PST 2021
;; MSG SIZE  rcvd: 54

[email protected]:~ $ dig +short CHAOS TXT id.server @1.1.1.1
"SAN"
[email protected]:~ $ dig +short CHAOS TXT id.server @1.0.0.1
"SAN"
[email protected]:~ $ traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
 1  <redacted>  0.375 ms  0.285 ms  0.271 ms
 2  <redacted>  2.872 ms  2.792 ms  2.712 ms
 3  et-4-0-0-500.edge7.LosAngeles1.Level3.net (4.16.27.109)  8.684 ms  8.580 ms  8.464 ms
 4  ae-1-3.bar1.SanDiego1.Level3.net (4.69.140.101)  11.471 ms  11.366 ms  11.260 ms
 5  4.30.5.78 (4.30.5.78)  11.347 ms  11.042 ms  11.120 ms
 6  one.one.one.one (1.1.1.1)  10.801 ms  11.266 ms  10.978 ms
[email protected]:~ $ traceroute 1.0.0.1
traceroute to 1.0.0.1 (1.0.0.1), 30 hops max, 60 byte packets
 1  <redacted>  0.413 ms  0.280 ms  0.257 ms
 2  <redacted>  2.621 ms  3.014 ms  2.883 ms
 3  et-4-0-0-500.edge7.LosAngeles1.Level3.net (4.16.27.109)  8.516 ms  8.379 ms  8.255 ms
 4  ae-1-3.bar1.SanDiego1.Level3.net (4.69.140.101)  10.406 ms  10.275 ms *
 5  4.30.5.78 (4.30.5.78)  10.783 ms  10.656 ms  10.623 ms
 6  * one.one.one.one (1.0.0.1)  11.156 ms  46.550 ms
[email protected]:~ $ dig +short CHAOS TXT id.server @1.1.1.1
"SAN"
[email protected]:~ $ dig +short CHAOS TXT id.server @1.0.0.1
"SAN"
[email protected]:~ $ dig +tcp @1.1.1.1 id.server CH TXT

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> +tcp @1.1.1.1 id.server CH TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57691
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;id.server.                     CH      TXT

;; ANSWER SECTION:
id.server.              0       CH      TXT     "SAN"

;; Query time: 10 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Feb 25 08:23:46 PST 2021
;; MSG SIZE  rcvd: 43

[email protected]:~ $ dig +tcp @1.0.0.1 id.server CH TXT

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> +tcp @1.0.0.1 id.server CH TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35337
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;id.server.                     CH      TXT

;; ANSWER SECTION:
id.server.              0       CH      TXT     "SAN"

;; Query time: 45 msec
;; SERVER: 1.0.0.1#53(1.0.0.1)
;; WHEN: Thu Feb 25 08:23:47 PST 2021
;; MSG SIZE  rcvd: 43

[email protected]:~ $ curl -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=cloudflare.com&type=AAAA'
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":true,"CD":false,"Question":[{"name":"cloudflare.com","type":28}],"Answer":[{"name":"cloudflare.com","type":28,"TTL":255,"data":"2606:4700::6810:85e5"},{"name":"cloudflare.com","type":28,"TTL":255,"data":"2606:4700::6810:84e5"}]}

[sdayman]

That’s certainly odd. I’m also using Pi-hole with cloudflared that connects to LAX and we’re on Webex all day long. It’s strange that your command line shows SAN though. I’m closest to LAX but sometimes get routed to SJC.

[darren_ata]

Indeed, that’s my setup as well - Pi-hole with cloudflared running DoH to 1.1.1.1 & 1.0.0.1. Been that way for years now without issues. This just started yesterday morning and only appears to be affecting WebEx services. I switched my Pi-holes to bypass cloudflared and just use 8.8.8.8 until I can get this resolved as our household uses WebEx every day for work.

[anb]

Hi @darren_ata ,

Thank you for the report! The line ; OPT=15: 00 16 ("..") in dig output shows there was an extended DNS error 0x16(or 22), which is No Reachable Authority, meaning our resolver cannot exchange with upstream authoritative nameserver. In this case, it’s ns{1,2}.as13445.net.. I applied a workaround in SAN, and will notify the operator at as13445.net.

Let me know if you still have issue.

[darren_ata]

Thanks, it appears to be resolving now:

[email protected]:~ $ dig webex.com @1.1.1.1                                                                               
; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> webex.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51398
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;webex.com.                     IN      A

;; ANSWER SECTION:
webex.com.              74      IN      A       64.68.121.205

;; Query time: 14 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Feb 25 09:42:26 PST 2021
;; MSG SIZE  rcvd: 54

[email protected]:~ $ dig webex.com @1.0.0.1

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> webex.com @1.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10915
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;webex.com.                     IN      A

;; ANSWER SECTION:
webex.com.              224     IN      A       64.68.121.205

;; Query time: 15 msec
;; SERVER: 1.0.0.1#53(1.0.0.1)
;; WHEN: Thu Feb 25 09:42:37 PST 2021
;; MSG SIZE  rcvd: 54

I’ll be switching back over to cloudflared DoH after the work day is over and verify “in production” as well.

[eag082002]

The issue is resolved for me as well. Thank you!

[mfura6579]

Still down for me on the east coast. Had to switch to Google in order to get Webex to resolve.

{c:\BIND>dig @1.1.1.1 webex.com SOA

; <<>> DiG 9.14.11 <<>> @1.1.1.1 webex.com SOA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 13213
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 16 ("…")
;; QUESTION SECTION:
;webex.com. IN SOA

;; Query time: 3952 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Feb 26 01:34:57 Eastern Standard Time 2021
;; MSG SIZE rcvd: 44}

[sdayman]

Don’t forget dig +short CHAOS TXT id.server @1.1.1.1 so @anb knows which datacenter that’s in.

[mfura6579]

Thanks for that. Looks like it’s the Pittsburgh one?

c:\BIND>dig CHAOS TXT id.server @1.1.1.1 webex.com +short
“PIT”