, DNS over TLS limitations


I receive a REFUSED code from the and servers when I make numerous DNS queries via DoT from a DigitalOcean VDS server. Could you please explain this behavior and provide information about any limitations?

Below, you will find a detailed research story.

Over the past three weeks, I conducted research regarding a problem with the mobile app “Aviasales” and I want to share the results. I use Outline VPN with my own server on DigitalOcean and noticed that sometimes I experience network errors while using the Aviasales app when my VPN client is active. Here is a sample from the app logs:

metrics: NSURLSessionTaskMetrics(),
  underlyingError: NSError(
    domain: "NSURLErrorDomain",
    code: -1003,
    userInfo: [
      "NSErrorFailingURLKey": URL(https://mobile-explore.aviasales.com/personalization/v1/ios/aviasales/search/destination_suggestions.json?locale=en_TH),
      "NSErrorFailingURLStringKey": "https://mobile-explore.aviasales.com/personalization/v1/ios/aviasales/search/destination_suggestions.json?locale=en_TH",
      "NSLocalizedDescription": "A server with the specified hostname could not be found.",
      "NSUnderlyingError": NSError#2(
        domain: "kCFErrorDomainCFNetwork",
        code: -1003,

It seems strange because I had a browser open with many tabs, and the Internet was working well. I asked the developers of the app about this issue, but they found nothing wrong on their side. So, I concluded that the problem was on my end and likely related to DNS or network issues.

I found a way to reproduce the problem on a Mac laptop. When the issue occurred again, I tried to check the DNS name resolution using the dig utility, but the resolution worked normally.

dig mobile-explore.aviasales.com                                                                                 

; <<>> DiG 9.10.6 <<>> mobile-explore.aviasales.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12966
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

; EDNS: version: 0, flags:; udp: 1232
;mobile-explore.aviasales.com.	IN	A

mobile-explore.aviasales.com. 54 IN	A
mobile-explore.aviasales.com. 54 IN	A
mobile-explore.aviasales.com. 54 IN	A
mobile-explore.aviasales.com. 54 IN	A

;; Query time: 266 msec
;; WHEN: Wed Apr 17 17:39:00 +07 2024
;; MSG SIZE  rcvd: 121

I received a response from, but my laptop was configured to use the DNS server. I checked the source code of the Outline VPN client and found that it has hardcoded DNS addresses of public resolvers, with the first resolver being

I decided to capture VPN traffic with Wireshark and found that there were only a few packets on port 53, but many packets to and on port 853. It seems like MacOS activated DNS over TLS when the VPN extension was turned on.

I reproduced the problem several times and each time I examined the traffic dump and didn’t find anything unusual like TCP RST, a lot of retransmits, or significant latency. I saw that my system sent requests to the (sometimes server and received some responses, but I was unable to decrypt the traffic to analyze the responses.

I tried to set up a MitM proxy on my VPN server, but for some reason, MacOS was unable to use my own server for DoT working via standard UDP protocol and port 53. I was unable to reproduce the problem with common DNS protocol.

In MacOS, all DNS functions are handled by mDNSResponder. I checked its logs and found that they too were encrypted; I was only able to see DNS flags and system messages, but queries, responses, and servers were hashed:

2024-04-17 18:08:08.753948+0700 0x1beee7   Default     0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q388] Sent 128-byte query #1 to BBhjVjYR over TLS via utun7/32 -- id: 0x87F5 (34805), flags: 0x0100 (Q/Query, RD, NoError), counts: 1/0/0/1, BBPdIEdC IN A?, . OPT 512 0 {EDE, code: 0 (Other Error), extra-text: ''}, {Padding, [76 B]}

One day, I discovered a correlation between the errors and a large volume of DNS queries (during the tests, I frequently cleared the system DNS cache and restarted various apps to identify the problem). I hypothesized that the problem was related to some IP-based limitation. My VPN server has only one IP, which can generate tons of DNS traffic. I wrote a small Go application to resolve about 120 domains and encountered resolution errors in case of problems.

domains := ["lot of different domains here like cloudflare.com,google.com,etc"]

for {  
    fmt.Printf("Number of domains: %d\n", len(domains))  
    randomTime := time.Duration(rand.Intn(15) + 1)  
    for _, domain := range domains {  
       addr, err := net.LookupHost(domain)  
       if err != nil {  
          fmt.Printf("Timestamp: %s Error with domain %s: %v\n", time.Now(), domain, err)  
       fmt.Printf("Domain %s have address: %v\n", domain, addr)  
    fmt.Printf("Next loop in %d sec\n", randomTime)  
    time.Sleep(randomTime * time.Second)  

After running my program via VPN for a few seconds, I encountered a DNS resolution error like this:

Domain phasmophobia.com have address: []
Timestamp: 2024-04-17 18:13:31.97806 +0700 +07 m=+26.222307001 Error with domain rust.com: lookup rust.com: no such host

In the mDNSResponder logs, I found the following:

2024-04-17 18:13:31.573270+0700 0x1c0536   Default     0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q61011] Received acceptable 468-byte response from BBhjVjYR over TLS via utun7/32 -- id: 0x0E9D (3741), flags: 0x8185 (R/Query, RD, RA, Refused), counts: 1/0/0/1, BBVZaOEV IN A?, . OPT 1232 0 {Padding, [427 B]}
2024-04-17 18:13:31.573407+0700 0x1c0536   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] Penalizing server BBhjVjYR for 60 seconds
2024-04-17 18:13:31.573488+0700 0x1c0536   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q61011] Querier concluded -- reason: response
2024-04-17 18:13:31.573586+0700 0x1c06af   Default     0x0                  389    0    mDNSResponder: [com.apple.mDNSResponder:Default] [Q61011] Handling concluded querier: BBVZaOEV A IN
2024-04-17 18:13:31.573737+0700 0x1c06af   Default     0x0                  389    0    mDNSResponder: [com.apple.mDNSResponder:Default] [R73761->Q61011] DNSServiceQueryRecord(<mask.hash: '3Inv9cAsh+PN1OHMZoeIhw=='>(d04a5b71), A) RESULT ADD interface 0: (mortal, DNSSEC Indeterminate)<mask.hash: '5PfwLdOJK8f/Zrbtqqno5w=='>
2024-04-17 18:13:31.976371+0700 0x1c06af   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] Unpenalizing responsive server BBhjVjYR
2024-04-17 18:13:31.976426+0700 0x1c06af   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q4131] Querier concluded -- reason: response

Although I still couldn’t see the queries and responses, I could see the flags:

R/Query, RD, RA, Refused

It seems I received a response code of 5 = REFUSED, and the system resolver temporarily disabled resolving for this server.

I am now able to reproduce this error consistently. I have encountered errors through different DigitalOcean VDS servers (though it seems to occur infrequently and may depend on the volume of traffic). I can also reproduce it via your app WARP (with Outline connected to my server, of course).

The main question is: how can I avoid these limits? We have many tickets from mobile app users who use it with Outline VPN and experience the same problems. It appears to be a major issue for anyone using a VPN.

If you need some details, I can provide VPN sever address, traffic dumps etc


Hi there? Any information about it?

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.