1.1.1.1,1.0.0.1 DNS over TLS limitations

Hello!

I receive a REFUSED code from the 1.1.1.1 and 1.0.0.1 servers when I make numerous DNS queries via DoT from a DigitalOcean VDS server. Could you please explain this behavior and provide information about any limitations?

Below, you will find a detailed research story.

Over the past three weeks, I conducted research regarding a problem with the mobile app “Aviasales” and I want to share the results. I use Outline VPN with my own server on DigitalOcean and noticed that sometimes I experience network errors while using the Aviasales app when my VPN client is active. Here is a sample from the app logs:

metrics: NSURLSessionTaskMetrics(),
  underlyingError: NSError(
    domain: "NSURLErrorDomain",
    code: -1003,
    userInfo: [
      "NSErrorFailingURLKey": URL(https://mobile-explore.aviasales.com/personalization/v1/ios/aviasales/search/destination_suggestions.json?locale=en_TH),
      "NSErrorFailingURLStringKey": "https://mobile-explore.aviasales.com/personalization/v1/ios/aviasales/search/destination_suggestions.json?locale=en_TH",
      "NSLocalizedDescription": "A server with the specified hostname could not be found.",
      "NSUnderlyingError": NSError#2(
        domain: "kCFErrorDomainCFNetwork",
        code: -1003,

It seems strange because I had a browser open with many tabs, and the Internet was working well. I asked the developers of the app about this issue, but they found nothing wrong on their side. So, I concluded that the problem was on my end and likely related to DNS or network issues.

I found a way to reproduce the problem on a Mac laptop. When the issue occurred again, I tried to check the DNS name resolution using the dig utility, but the resolution worked normally.

dig mobile-explore.aviasales.com                                                                                 

; <<>> DiG 9.10.6 <<>> mobile-explore.aviasales.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12966
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;mobile-explore.aviasales.com.	IN	A

;; ANSWER SECTION:
mobile-explore.aviasales.com. 54 IN	A	99.86.4.92
mobile-explore.aviasales.com. 54 IN	A	99.86.4.22
mobile-explore.aviasales.com. 54 IN	A	99.86.4.106
mobile-explore.aviasales.com. 54 IN	A	99.86.4.102

;; Query time: 266 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Wed Apr 17 17:39:00 +07 2024
;; MSG SIZE  rcvd: 121

I received a response from 1.1.1.1, but my laptop was configured to use the 8.8.8.8 DNS server. I checked the source code of the Outline VPN client and found that it has hardcoded DNS addresses of public resolvers, with the first resolver being 1.1.1.1.

I decided to capture VPN traffic with Wireshark and found that there were only a few packets on port 53, but many packets to 1.1.1.1 and 1.0.0.1 on port 853. It seems like MacOS activated DNS over TLS when the VPN extension was turned on.

I reproduced the problem several times and each time I examined the traffic dump and didn’t find anything unusual like TCP RST, a lot of retransmits, or significant latency. I saw that my system sent requests to the 1.1.1.1 (sometimes 1.0.0.1) server and received some responses, but I was unable to decrypt the traffic to analyze the responses.

I tried to set up a MitM proxy on my VPN server, but for some reason, MacOS was unable to use my own server for DoT working via standard UDP protocol and port 53. I was unable to reproduce the problem with common DNS protocol.

In MacOS, all DNS functions are handled by mDNSResponder. I checked its logs and found that they too were encrypted; I was only able to see DNS flags and system messages, but queries, responses, and servers were hashed:

2024-04-17 18:08:08.753948+0700 0x1beee7   Default     0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q388] Sent 128-byte query #1 to BBhjVjYR over TLS via utun7/32 -- id: 0x87F5 (34805), flags: 0x0100 (Q/Query, RD, NoError), counts: 1/0/0/1, BBPdIEdC IN A?, . OPT 512 0 {EDE, code: 0 (Other Error), extra-text: ''}, {Padding, [76 B]}

One day, I discovered a correlation between the errors and a large volume of DNS queries (during the tests, I frequently cleared the system DNS cache and restarted various apps to identify the problem). I hypothesized that the problem was related to some IP-based limitation. My VPN server has only one IP, which can generate tons of DNS traffic. I wrote a small Go application to resolve about 120 domains and encountered resolution errors in case of problems.

domains := ["lot of different domains here like cloudflare.com,google.com,etc"]

for {  
    fmt.Printf("Number of domains: %d\n", len(domains))  
    randomTime := time.Duration(rand.Intn(15) + 1)  
    for _, domain := range domains {  
       addr, err := net.LookupHost(domain)  
       if err != nil {  
          fmt.Printf("Timestamp: %s Error with domain %s: %v\n", time.Now(), domain, err)  
          return  
       }  
       fmt.Printf("Domain %s have address: %v\n", domain, addr)  
    }  
    fmt.Printf("Next loop in %d sec\n", randomTime)  
    time.Sleep(randomTime * time.Second)  
}

After running my program via VPN for a few seconds, I encountered a DNS resolution error like this:

Domain phasmophobia.com have address: [13.248.169.48 76.223.54.146]
Timestamp: 2024-04-17 18:13:31.97806 +0700 +07 m=+26.222307001 Error with domain rust.com: lookup rust.com: no such host

In the mDNSResponder logs, I found the following:

2024-04-17 18:13:31.573270+0700 0x1c0536   Default     0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q61011] Received acceptable 468-byte response from BBhjVjYR over TLS via utun7/32 -- id: 0x0E9D (3741), flags: 0x8185 (R/Query, RD, RA, Refused), counts: 1/0/0/1, BBVZaOEV IN A?, . OPT 1232 0 {Padding, [427 B]}
2024-04-17 18:13:31.573407+0700 0x1c0536   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] Penalizing server BBhjVjYR for 60 seconds
2024-04-17 18:13:31.573488+0700 0x1c0536   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q61011] Querier concluded -- reason: response
2024-04-17 18:13:31.573586+0700 0x1c06af   Default     0x0                  389    0    mDNSResponder: [com.apple.mDNSResponder:Default] [Q61011] Handling concluded querier: BBVZaOEV A IN
2024-04-17 18:13:31.573737+0700 0x1c06af   Default     0x0                  389    0    mDNSResponder: [com.apple.mDNSResponder:Default] [R73761->Q61011] DNSServiceQueryRecord(<mask.hash: '3Inv9cAsh+PN1OHMZoeIhw=='>(d04a5b71), A) RESULT ADD interface 0: (mortal, DNSSEC Indeterminate)<mask.hash: '5PfwLdOJK8f/Zrbtqqno5w=='>
2024-04-17 18:13:31.976371+0700 0x1c06af   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] Unpenalizing responsive server BBhjVjYR
2024-04-17 18:13:31.976426+0700 0x1c06af   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q4131] Querier concluded -- reason: response

Although I still couldn’t see the queries and responses, I could see the flags:

R/Query, RD, RA, Refused

It seems I received a response code of 5 = REFUSED, and the system resolver temporarily disabled resolving for this server.

I am now able to reproduce this error consistently. I have encountered errors through different DigitalOcean VDS servers (though it seems to occur infrequently and may depend on the volume of traffic). I can also reproduce it via your app WARP (with Outline connected to my server, of course).

The main question is: how can I avoid these limits? We have many tickets from mobile app users who use it with Outline VPN and experience the same problems. It appears to be a major issue for anyone using a VPN.

If you need some details, I can provide VPN sever address, traffic dumps etc

4 Likes

Hi there? Any information about it?

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.