1.1.1.1 unable to resolve MX of SFDC, only in Toronto area


#1

Hi There,

I’m having an issue where I am unable to resolve MX records of SalesForce, but this is only happening in the Greater Toronto Area. There doesn’t appear to be issues with this in Oregon, USA and in the UK.

Here’s the result I’m getting in Toronto on Cogeco and Rogers ISPs.
> salesforce.com
Server: 1dot1dot1dot1.cloudflare-dns.com
Address: 1.1.1.1

*** 1dot1dot1dot1.cloudflare-dns.com can't find salesforce.com: Server failed

Yet no issues when I try in Oregon or UK.

salesforce.com
Server:  1dot1dot1dot1.cloudflare-dns.com
Address:  1.1.1.1

Non-authoritative answer:
salesforce.com  MX preference = 1, mail exchanger = aspmx.l.google.com
salesforce.com  MX preference = 5, mail exchanger = alt1.aspmx.l.google.com
salesforce.com  MX preference = 5, mail exchanger = alt2.aspmx.l.google.com
salesforce.com  MX preference = 10, mail exchanger = alt3.aspmx.l.google.com
salesforce.com  MX preference = 10, mail exchanger = alt4.aspmx.l.google.com

Is there a reason why this location isn’t able to resolve the MX records?


#2

Try following this guide, but this is probably something @cscharff has to look at since I am nowhere near there.

Have problems with 1.1.1.1? *Read Me First*


#3

—Edited, had the wrong dig syntax. Now updated

It’s only SalesFoce (and tertiary domains under it) that is having resolution problems from what I can tell. Normal resolutions, including MX lookups for other domains are working as expected.

 dig @1.1.1.1 salesforce.com mx

; <<>> DiG 9.11.3-RedHat-9.11.3-4.fc27 <<>> @1.1.1.1 salesforce.com mx
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 40947
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1536
;; QUESTION SECTION:
;salesforce.com.                        IN      MX

;; Query time: 2 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue May 01 09:40:04 EDT 2018
;; MSG SIZE  rcvd: 43

Here is the result on a different DNS provider.

dig @208.91.112.52 salesforce.com mx

; <<>> DiG 9.11.3-RedHat-9.11.3-4.fc27 <<>> @208.91.112.52 salesforce.com mx
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27720
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 10, ADDITIONAL: 20

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;salesforce.com.                        IN      MX

;; ANSWER SECTION:
salesforce.com.         2227    IN      MX      10 alt4.aspmx.l.google.com.
salesforce.com.         2227    IN      MX      5 alt1.aspmx.l.google.com.
salesforce.com.         2227    IN      MX      1 aspmx.l.google.com.
salesforce.com.         2227    IN      MX      5 alt2.aspmx.l.google.com.
salesforce.com.         2227    IN      MX      10 alt3.aspmx.l.google.com.

;; AUTHORITY SECTION:
salesforce.com.         16235   IN      NS      ns3.salesforce.com.
salesforce.com.         16235   IN      NS      dns05.salesforce.com.
salesforce.com.         16235   IN      NS      ns4.salesforce.com.
salesforce.com.         16235   IN      NS      ns1.salesforce.com.
salesforce.com.         16235   IN      NS      dns02.salesforce.com.
salesforce.com.         16235   IN      NS      dns03.salesforce.com.
salesforce.com.         16235   IN      NS      dns06.salesforce.com.
salesforce.com.         16235   IN      NS      dns01.salesforce.com.
salesforce.com.         16235   IN      NS      ns2.salesforce.com.
salesforce.com.         16235   IN      NS      dns04.salesforce.com.

;; ADDITIONAL SECTION:
alt3.aspmx.l.google.com. 63     IN      A       173.194.79.27
alt4.aspmx.l.google.com. 63     IN      A       108.177.14.27
alt1.aspmx.l.google.com. 179    IN      A       209.85.203.27
alt1.aspmx.l.google.com. 272    IN      AAAA    2800:3f0:4003:c00::1a
alt2.aspmx.l.google.com. 240    IN      A       209.85.203.26
alt2.aspmx.l.google.com. 239    IN      AAAA    2a00:1450:400b:c03::1b
aspmx.l.google.com.     266     IN      A       173.194.175.26
dns02.salesforce.com.   40268   IN      A       204.74.109.235
dns06.salesforce.com.   40268   IN      A       204.74.115.235
dns01.salesforce.com.   40268   IN      A       204.74.108.235
dns05.salesforce.com.   63423   IN      A       204.74.114.235
ns2.salesforce.com.     40268   IN      A       204.13.250.39
ns1.salesforce.com.     54583   IN      A       208.78.70.39
ns1.salesforce.com.     100143  IN      AAAA    2001:500:90:1::39
ns3.salesforce.com.     54583   IN      A       208.78.71.39
ns3.salesforce.com.     100143  IN      AAAA    2001:500:94:1::39
dns04.salesforce.com.   43448   IN      A       199.7.69.235
ns4.salesforce.com.     8954    IN      A       204.13.251.39
dns03.salesforce.com.   40268   IN      A       199.7.68.235

;; Query time: 54 msec
;; SERVER: 208.91.112.52#53(208.91.112.52)
;; WHEN: Tue May 01 09:40:23 EDT 2018
;; MSG SIZE  rcvd: 702

I do not have a linux box in other locations to do the dig, but since they are working in those areas, I do not believe it’ll be necessary.


#4

Here’s examples of tertiary domains of SalesForce which works in other locations, but not in the Toronto area.

dig @1.1.1.1 test.in.salesforce.com mx

; <<>> DiG 9.11.3-RedHat-9.11.3-4.fc27 <<>> @1.1.1.1 test.in.salesforce.com mx
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 64200
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1536
;; QUESTION SECTION:
;test.in.salesforce.com.                IN      MX

;; Query time: 3 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue May 01 09:43:53 EDT 2018
;; MSG SIZE  rcvd: 51

Again, with a different provider.

dig @208.91.112.52 test.in.salesforce.com mx

; <<>> DiG 9.11.3-RedHat-9.11.3-4.fc27 <<>> @208.91.112.52 test.in.salesforce.com mx
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20337
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 10, ADDITIONAL: 13

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;test.in.salesforce.com.                IN      MX

;; ANSWER SECTION:
test.in.salesforce.com. 300     IN      MX      10 mx3-was3.mta.salesforce.com.
test.in.salesforce.com. 300     IN      MX      10 mx1-chi3.mta.salesforce.com.
test.in.salesforce.com. 300     IN      MX      10 mx1-was3.mta.salesforce.com.
test.in.salesforce.com. 300     IN      MX      10 mx4-chi3.mta.salesforce.com.
test.in.salesforce.com. 300     IN      MX      10 mx2-was3.mta.salesforce.com.
test.in.salesforce.com. 300     IN      MX      10 mx2-chi3.mta.salesforce.com.
test.in.salesforce.com. 300     IN      MX      10 mx4-was3.mta.salesforce.com.
test.in.salesforce.com. 300     IN      MX      10 mx3-chi3.mta.salesforce.com.

;; AUTHORITY SECTION:
salesforce.com.         16046   IN      NS      dns06.salesforce.com.
salesforce.com.         16046   IN      NS      dns02.salesforce.com.
salesforce.com.         16046   IN      NS      ns4.salesforce.com.
salesforce.com.         16046   IN      NS      ns3.salesforce.com.
salesforce.com.         16046   IN      NS      dns04.salesforce.com.
salesforce.com.         16046   IN      NS      ns2.salesforce.com.
salesforce.com.         16046   IN      NS      dns03.salesforce.com.
salesforce.com.         16046   IN      NS      dns05.salesforce.com.
salesforce.com.         16046   IN      NS      dns01.salesforce.com.
salesforce.com.         16046   IN      NS      ns1.salesforce.com.

;; ADDITIONAL SECTION:
dns02.salesforce.com.   40079   IN      A       204.74.109.235
dns06.salesforce.com.   40079   IN      A       204.74.115.235
dns01.salesforce.com.   40079   IN      A       204.74.108.235
dns05.salesforce.com.   63234   IN      A       204.74.114.235
ns2.salesforce.com.     40079   IN      A       204.13.250.39
ns1.salesforce.com.     54394   IN      A       208.78.70.39
ns1.salesforce.com.     99954   IN      AAAA    2001:500:90:1::39
ns3.salesforce.com.     54394   IN      A       208.78.71.39
ns3.salesforce.com.     99954   IN      AAAA    2001:500:94:1::39
dns04.salesforce.com.   43259   IN      A       199.7.69.235
ns4.salesforce.com.     8765    IN      A       204.13.251.39
dns03.salesforce.com.   40079   IN      A       199.7.68.235

;; Query time: 60 msec
;; SERVER: 208.91.112.52#53(208.91.112.52)
;; WHEN: Tue May 01 09:43:32 EDT 2018
;; MSG SIZE  rcvd: 663

#5

I thought I added this note, but it looks like I didn’t. This only started happening as of Monday (yesterday) around 2:30pm EDT. There wasn’t issues before that time as we constantly send emails to SalesForce from our organization. The problem came to a surprise to us and took us awhile to track it down to being a Cloudflare issue.


#6

@jason.chambers thanks for the report. I’ll open an issue with my team now. Great troubleshooting, very helpful. Would you mind running this command as well where you’re failing so I can be sure exacrtly which Cloudflare POP you’re hitting?

dig +short CHAOS TXT id.server @1.1.1.1

And if possible would you try doing the same Mx lookup against 1.0.0.1 to see if you get the same results? (and the above command against it as well?


#7

No problem. Yes, 1.0.0.1 is returning the same results as well.

dig @1.0.0.1 salesforce.com mx

; <<>> DiG 9.11.3-RedHat-9.11.3-4.fc27 <<>> @1.0.0.1 salesforce.com mx
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 56363
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1536
;; QUESTION SECTION:
;salesforce.com.                        IN      MX

;; Query time: 3 msec
;; SERVER: 1.0.0.1#53(1.0.0.1)
;; WHEN: Tue May 01 10:14:58 EDT 2018
;; MSG SIZE  rcvd: 43

The server id is yyz01 for both results. (cute, you’re using airport icao codes).


#8

I think I was able to repro the problem from YYZ despite the Rush flashback. Not sure of the exact root cause, but I’ve added soem additional data along with the info you provided to the ticket.


#9

@jason.chambers would you mind testing again when you get a minute? We’re in the process of reverting a change in that POP.


#10

Seems to be working now.

> set type=mx
> salesforce.com
Server:  1dot1dot1dot1.cloudflare-dns.com
Address:  1.1.1.1

Non-authoritative answer:
salesforce.com  MX preference = 1, mail exchanger = aspmx.l.google.com
salesforce.com  MX preference = 5, mail exchanger = alt1.aspmx.l.google.com
salesforce.com  MX preference = 5, mail exchanger = alt2.aspmx.l.google.com
salesforce.com  MX preference = 10, mail exchanger = alt3.aspmx.l.google.com
salesforce.com  MX preference = 10, mail exchanger = alt4.aspmx.l.google.com
> test.in.salesforce.com
Server:  1dot1dot1dot1.cloudflare-dns.com
Address:  1.1.1.1

Non-authoritative answer:
test.in.salesforce.com  MX preference = 10, mail exchanger = mx1-chi3.mta.salesforce.com
test.in.salesforce.com  MX preference = 10, mail exchanger = mx1-was3.mta.salesforce.com
test.in.salesforce.com  MX preference = 10, mail exchanger = mx2-chi3.mta.salesforce.com
test.in.salesforce.com  MX preference = 10, mail exchanger = mx2-was3.mta.salesforce.com
test.in.salesforce.com  MX preference = 10, mail exchanger = mx3-chi3.mta.salesforce.com
test.in.salesforce.com  MX preference = 10, mail exchanger = mx3-was3.mta.salesforce.com
test.in.salesforce.com  MX preference = 10, mail exchanger = mx4-chi3.mta.salesforce.com
test.in.salesforce.com  MX preference = 10, mail exchanger = mx4-was3.mta.salesforce.com

Hopefully it was an easy fix for you guys!


#11

We reverted a change that removed workarounds for non-compliant F5 load balancer in Salesforce’s DNS infra, it should be working now.


#12

This topic was automatically closed after 14 days. New replies are no longer allowed.