DoH error w/ cloudflared-service

After I day of troubleshooting I can’t seem to address the issue below:

Dec 12 09:03:52 mintvm cloudflared[5706]: time=“2019-12-12T09:03:52-06:00” level=error msg=“failed to connect to an HTTPS backend "https://1.1.1.1/dns-query\”" error=“failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: x509: cannot validate certificate for 1.1.1.1 because it doesn’t contain any IP SANs”

any suggestions or direction?

Thanks

Are you sure you are actually reaching Cloudflare and they address hasnt been hijacked by your ISP? The latter would seem most likely in this case.

What does a traceroute to 1.1.1.1 say?

Do you also have issues with 1.0.0.1?

2 Likes

Yes, it’s both 1.1.1.1 and 1.0.0.1

traceroute to 1.1.1.1 (1.1.1.1), 64 hops max
1 192.168.1.1 0.951ms 0.575ms 0.512ms
2 x.x.x.x 13.294ms 13.973ms 10.064ms
3 x.x.x.x 13.542ms 9.263ms 16.162ms
4 x.x.x.x 17.711ms 13.876ms 14.671ms
5 x.x.x.x 16.519ms 15.260ms 14.436ms
6 1.1.1.1 16.219ms 14.473ms 15.167ms
guyp@mintvm:~$ traceroute 1.0.0.1
traceroute to 1.0.0.1 (1.0.0.1), 64 hops max
1 192.168.1.1 0.864ms 0.638ms 0.615ms
2 x.x.x.x 11.873ms 9.650ms 11.503ms
3 x.x.x.x 11.594ms 10.502ms 12.996ms
4 x.x.x.x 18.051ms 15.343ms 15.041ms
5 x.x.x.x 17.405ms 17.677ms 18.375ms
6 1.0.0.1 16.878ms 14.431ms 15.688ms

Thanks

Censoring debug output is not really helpful. Post the actual traceroutes. But it would seem very much as if your ISP hijacked the addresses. You should probably contact your ISP.

My ISP is comcast and if I use 1.1.1.1 as my normal DNS no issues. Below is my config and dig command if that is helpful:

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> @127.0.0.1 -p 5053 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 37665
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 611e4ff4503dc91e (echoed)
;; QUESTION SECTION:
;google.com. IN A

;; Query time: 241 msec
;; SERVER: 127.0.0.1#5053(127.0.0.1)
;; WHEN: Thu Dec 12 09:56:13 CST 2019
;; MSG SIZE rcvd: 51

CLOUDFLARED_OPTS=–port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query

Thanks

You need to post the traceroute.

See above.

Thanks

That does seem as if your ISP hijacked these addresses. All of that is still within their networks.

What do these commands return?

dig +short CHAOS TXT id.server @1.1.1.1
dig @1.1.1.1 cloudflare.com
openssl s_client -connect 1.1.1.1:443

Closed

Well, yes you do seem to connect to Cloudflare, however the certificate does return the right configuration as well.

-----BEGIN CERTIFICATE-----
MIIFxjCCBUygAwIBAgIQAczjGN6fVn+rKySQH62nHTAKBggqhkjOPQQDAjBMMQsw
CQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSYwJAYDVQQDEx1EaWdp
Q2VydCBFQ0MgU2VjdXJlIFNlcnZlciBDQTAeFw0xOTAxMjgwMDAwMDBaFw0yMTAy
MDExMjAwMDBaMHIxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYw
FAYDVQQHEw1TYW4gRnJhbmNpc2NvMRkwFwYDVQQKExBDbG91ZGZsYXJlLCBJbmMu
MRswGQYDVQQDExJjbG91ZGZsYXJlLWRucy5jb20wWTATBgcqhkjOPQIBBggqhkjO
PQMBBwNCAATFIHCMIEJQKB59REF8MHkpHGNeHUSbxfdxOive0qKksWw9ash3uMuP
LlBT/fQYJn9hN+3/wr7pC125fuHfHOJ0o4ID6DCCA+QwHwYDVR0jBBgwFoAUo53m
H/naOU/AbuiRy5Wl2jHiCp8wHQYDVR0OBBYEFHCV3FyjjmYH28uBEMar58OoRX+g
MIGsBgNVHREEgaQwgaGCEmNsb3VkZmxhcmUtZG5zLmNvbYIUKi5jbG91ZGZsYXJl
LWRucy5jb22CD29uZS5vbmUub25lLm9uZYcEAQEBAYcEAQAAAYcEop+ENYcQJgZH
AEcAAAAAAAAAAAAREYcQJgZHAEcAAAAAAAAAAAAQAYcQJgZHAEcAAAAAAAAAAAAA
ZIcQJgZHAEcAAAAAAAAAAABkAIcEop8kAYcEop8uATAOBgNVHQ8BAf8EBAMCB4Aw
HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMGkGA1UdHwRiMGAwLqAsoCqG
KGh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9zc2NhLWVjYy1nMS5jcmwwLqAsoCqG
KGh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9zc2NhLWVjYy1nMS5jcmwwTAYDVR0g
BEUwQzA3BglghkgBhv1sAQEwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cuZGln
aWNlcnQuY29tL0NQUzAIBgZngQwBAgIwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUF
BzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6
Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEVDQ1NlY3VyZVNlcnZlckNB
LmNydDAMBgNVHRMBAf8EAjAAMIIBfgYKKwYBBAHWeQIEAgSCAW4EggFqAWgAdgCk
uQmQtBhYFIe7E6LMZ3AKPDWYBPkb37jjd80OyA3cEAAAAWiVHhSLAAAEAwBHMEUC
IQDlnoPeMXtFkRsy3Vs0eovk3ILKt01x6bgUdMlmQTFIvAIgcAn0lFSjiGzHm2eO
jDZJzMiP5Uaj0Jwub9GO8RkxkkoAdQCHdb/nWXz4jEOZX73zbv9WjUdWNv9KtWDB
tOr/XqCDDwAAAWiVHhVsAAAEAwBGMEQCIFC0n0JModeol8b/Qicxd5Blf/o7xOs/
Bk0j9hdc5N7jAiAQocYnHL9iMqTtFkh0vmSsII5NbiakM/2yDEXnwkPRvAB3ALvZ
37wfinG1k5Qjl6qSe0c4V5UKq1LoGpCWZDaOHtGFAAABaJUeFJEAAAQDAEgwRgIh
AL3OPTBzOZpS5rS/uLzqMOiACCFQyY+mTJ+L0I9TcB3RAiEA4+SiPz0/5kFxvrk7
AKYKdvelgV1hiiPbM2YHY+/0BIkwCgYIKoZIzj0EAwIDaAAwZQIwez76hX2HTMur
/I3XRuwfdmVoa8J6ZVEVq+AZsE7DyQh7AV4WNLU+092BrPbnyVUFAjEAzUf5jdz1
pyc74lgOunC7LBE6cPtWbzfGpJiYyT/T+c5eIAwRYziKT0DKbaql7tiZ
-----END CERTIFICATE-----

Could this be a FW issue going over port 5053??

How is a firewall and that port related here? The original error message only referred to the URL in question. If you do some rewriting or have some MITM setup, that could of course change things, but that would be down to your setup.

No proxy, just a VM running Linux Mint 19.2…

Then what about the firewall and port 5053?

The OpenSSL output is in stark contrast to what the original error said. It connects properly and does return the right certificate. Thats cloudflared in the first message, right?

Yes, it is cloudflared in the first message.

Thanks


Yup I think it was a FW issue and seems to be working now, doh!

root@mintvm:/etc# dig @127.0.0.1 -p 5053 google.com

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> @127.0.0.1 -p 5053 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12051
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
; PAD: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 (“…”)
;; QUESTION SECTION:
;google.com. IN A

;; ANSWER SECTION:
google.com. 289 IN A 64.233.177.101
google.com. 289 IN A 64.233.177.102
google.com. 289 IN A 64.233.177.113
google.com. 289 IN A 64.233.177.138
google.com. 289 IN A 64.233.177.139
google.com. 289 IN A 64.233.177.100

;; Query time: 16 msec
;; SERVER: 127.0.0.1#5053(127.0.0.1)
;; WHEN: Thu Dec 12 10:42:07 CST 2019
;; MSG SIZE rcvd: 316

That shouldnt be related to port 5053, however. If you had blocked cloudflared from accessing the Internet, it would certainly explain why it couldnt validate the certificate, because it couldnt connect in the first place, however if that was the case, the error message would be quite misleading. That wouldnt have been an invalid certificate but rather a timed out or reset connection.

Understood, to test I created a pass rule for the host until I can determine specifically what was/is occurring and setup the ACLs accordingly.

Thanks

This topic was automatically closed after 31 days. New replies are no longer allowed.