DNS over HTTPS not working Nov-26-2019

Today Nov-26–2019 all my DNs over HTTPs devices which include linux , Mac OS and Android stopped working over DOH.The cloudflare settings are as follows.

https://1.1.1.1/help#eyJpc0NmIjoiTm8iLCJpc0RvdCI6Ik5vIiwiaXNEb2giOiJObyIsInJlc29sdmVySXAtMS4xLjEuMSI6IlllcyIsInJlc29sdmVySXAtMS4wLjAuMSI6IlllcyIsInJlc29sdmVySXAtMjYwNjo0NzAwOjQ3MDA6OjExMTEiOiJObyIsInJlc29sdmVySXAtMjYwNjo0NzAwOjQ3MDA6OjEwMDEiOiJObyIsImRhdGFjZW50ZXJMb2NhdGlvbiI6IllZWiIsImlzV2FycCI6Ik5vIiwiaXNwTmFtZSI6IkNsb3VkZmxhcmUiLCJpc3BBc24iOiIxMzMzNSJ9

The traceroute shows access to the 1.1.1.1 network 3 3.52.251.198.in-addr.arpa (198.251.52.3) 20.584 ms 81.945 ms 56.215 ms

4 198.251.49.89 (198.251.49.89) 21.406 ms 21.144 ms 22.069 ms 5 198.251.51.56 (198.251.51.56) 22.771 ms 22.709 ms 21.188 ms 6 198.251.50.16 (198.251.50.16) 22.807 ms 78.695 ms 96.269 ms 7 cloudflare.ip4.torontointernetxchange.net (206.108.34.208) 106.362 ms 93.731 ms 95.215 ms 8 one.one.one.one (1.1.1.1) 164.851 ms * 88.710 ms

My Firefox setting has DNS over hTTPS enabled no extensions and configured for cloudflare and my about:config settings network.trr.bootstrapAddress;1.1.1.1 network.trr.mode;3 network.trr.resolvers;[{ “name”: “Cloudflare”, “url”: “https://mozilla.cloudflare-dns.com/dns-query” }]

Network mode 3 means DOH should only resolve using DOH but I can still get to other sites instead of a message say “hmm where having trouble finding that site”

Does anyone know why this is occurring?

Are you running Windows 10?
If so, what is the output of the following PowerShell command?

(Invoke-WebRequest -Uri 'https://1.1.1.1/dns-query?ct=application/dns-json&name=cloudflare.com').RawContent

I am running on a Mac and Android and Linux

In that case run

curl -v 'https://1.1.1.1/dns-query?ct=application/dns-json&name=cloudflare.com'

The result

  • Trying 1.1.1.1…
  • TCP_NODELAY set
  • Connected to 1.1.1.1 (1.1.1.1) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/cert.pem
    CApath: none
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS change cipher, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare-dns.com
  • start date: Jan 28 00:00:00 2019 GMT
  • expire date: Feb 1 12:00:00 2021 GMT
  • subjectAltName: host “1.1.1.1” matched cert’s IP address!
  • issuer: C=US; O=DigiCert Inc; CN=DigiCert ECC Secure Server CA
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x7fe24b008a00)

GET /dns-query?ct=application/dns-json&name=cloudflare.com HTTP/2
Host: 1.1.1.1
User-Agent: curl/7.54.0
Accept: /

  • Connection state changed (MAX_CONCURRENT_STREAMS updated)!
    < HTTP/2 400
    < date: Wed, 27 Nov 2019 03:34:27 GMT
    < content-length: 0
    < access-control-allow-origin: *
    < expect-ct: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
    < server: cloudflare
    < cf-ray: 53c10be5f882abac-YYZ
    <
  • Connection #0 to host 1.1.1.1 left intact

Interesting, you seem to get a 400 error.
That is most likely an issue restricted to the YYZ datacentre.

What if you try 1.0.0.1?

curl -v 'https://1.0.0.1/dns-query?ct=application/dns-json&name=cloudflare.com'

@dane @irtefa

What do you mean 400 error? The connectivity script shows access to both 1.1.1.1 and 1.0.0.1 also my traceroute is successful to 1.1.1.1

Post the output of the second command.

And while we are at it, lets try these too.

curl -v --http1.1 'https://1.1.1.1/dns-query?ct=application/dns-json&name=cloudflare.com'
curl -v --http1.1 'https://1.0.0.1/dns-query?ct=application/dns-json&name=cloudflare.com'

result 1
curl -v --http1.1 ‘https://1.1.1.1/dns-query?ct=application/dns-json&name=cloudflare.com

  • Trying 1.1.1.1…
  • TCP_NODELAY set
  • Connected to 1.1.1.1 (1.1.1.1) port 443 (#0)
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/cert.pem
    CApath: none
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS change cipher, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
  • ALPN, server accepted to use http/1.1
  • Server certificate:
  • subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare-dns.com
  • start date: Jan 28 00:00:00 2019 GMT
  • expire date: Feb 1 12:00:00 2021 GMT
  • subjectAltName: host “1.1.1.1” matched cert’s IP address!
  • issuer: C=US; O=DigiCert Inc; CN=DigiCert ECC Secure Server CA
  • SSL certificate verify ok.

GET /dns-query?ct=application/dns-json&name=cloudflare.com HTTP/1.1
Host: 1.1.1.1
User-Agent: curl/7.54.0
Accept: /

< HTTP/1.1 400 Bad Request
< Date: Wed, 27 Nov 2019 07:05:40 GMT
< Content-Length: 0
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Expect-CT: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
< Server: cloudflare
< CF-RAY: 53c241498ec6caa4-YYZ
<

  • Connection #0 to host 1.1.1.1 left intact

result 2
curl -v --http1.1 ‘https://1.0.0.1/dns-query?ct=application/dns-json&name=cloudflare.com

  • Trying 1.0.0.1…
  • TCP_NODELAY set
  • Connected to 1.0.0.1 (1.0.0.1) port 443 (#0)
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/cert.pem
    CApath: none
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS change cipher, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
  • ALPN, server accepted to use http/1.1
  • Server certificate:
  • subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare-dns.com
  • start date: Jan 28 00:00:00 2019 GMT
  • expire date: Feb 1 12:00:00 2021 GMT
  • subjectAltName: host “1.0.0.1” matched cert’s IP address!
  • issuer: C=US; O=DigiCert Inc; CN=DigiCert ECC Secure Server CA
  • SSL certificate verify ok.

GET /dns-query?ct=application/dns-json&name=cloudflare.com HTTP/1.1
Host: 1.0.0.1
User-Agent: curl/7.54.0
Accept: /

< HTTP/1.1 400 Bad Request
< Date: Wed, 27 Nov 2019 07:10:06 GMT
< Content-Length: 0
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Expect-CT: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
< Server: cloudflare
< CF-RAY: 53c247c7cf4ecab8-YYZ
<

  • Connection #0 to host 1.0.0.1 left intact
    MacBook-Pro:Documents mac$

You skipped the second command, but as the others all show the same response I assume that wont make a difference.

As I said in my earlier response this must be an issue local to YYZ and it would be best if you contacted support about it.

You mean my ISP vmedia? I connected via a VPN the Data center was NTT and the issue continues. I also looked at Firefox “about:networking#dns” and it shows a the following TRR values

Hostname Family TRR Addresses Expires (Seconds)
r4---sn-oguesnz6.googlevideo.com ipv4 true 74.125.171.250
1609
etc…

The TRR value for all sites is true which means Firefox is using DOH.

What is your reasoning? How would you explain the fault?

Did I say that? Please reread my original response from eleven hours ago. Only support can help you in this case.

You did not answer my question.

What is your reasoning? How would you explain the fault?

I only deal with my ISP Vmedia on which I am currently running public IP address 198.251.56.153. If there is a data center problem I should see the fault on Cloudflares `status page httsp://www.cloudflarestatus.com

And you didnt read my answer, otherwise your question would have been answered.

Apparently a local issue of YYZ, as I have said three times now.

You need to contact support, as I have also said three times now :wink:

Ideally, but not necessarily.

I am experiencing the same issue; DNS over HTTPS suddenly not working in the past 24h, where it was working previously. I’m also using the YYZ Cloudflare Data Center.

https://1.1.1.1/help#eyJpc0NmIjoiTm8iLCJpc0RvdCI6Ik5vIiwiaXNEb2giOiJObyIsInJlc29sdmVySXAtMS4xLjEuMSI6IlllcyIsInJlc29sdmVySXAtMS4wLjAuMSI6IlllcyIsInJlc29sdmVySXAtMjYwNjo0NzAwOjQ3MDA6OjExMTEiOiJObyIsInJlc29sdmVySXAtMjYwNjo0NzAwOjQ3MDA6OjEwMDEiOiJObyIsImRhdGFjZW50ZXJMb2NhdGlvbiI6IllZWiIsImlzV2FycCI6Ik5vIiwiaXNwTmFtZSI6IkNsb3VkZmxhcmUiLCJpc3BBc24iOiIxMzMzNSJ9

I’ve tried submitting a support ticket but am getting an HTTP 500 error:

API Request Failed: POST /api/v4/accounts/790850bba7535133396c6f4d98a2a560/support/ticket/create (500)

Is there some other way to follow up on this issue? Thanks in advance.

You also seem to go via YYZ, so it will be the same issue. You can also drop them an email to [email protected].

@dane @irtefa @cloonan

1 Like

You also seem to go via YYZ, so it will be the same issue.

I understand that, but without the ability to submit a support ticket, I don’t know how to escalate / notify Cloudflare of the issue. Can anyone else confirm problem submitting support tickets?

Edit: Ah, thanks, didn’t know about the email contact. Thanks!

Where did you initially make the support request to Cloudflare? I do not see any change in cloudflare status.

The IP address my workstations are using for DOH resolution is 108.162.240.42

You are referring to the error @mjsuhonos posted?