High DNS filtering latency with Cloudflare Gateway

Zero Trust Gateway
1

I’m using DNS Policies to block domains related to ads with Cloudflare Gateway.

I compared Gateway’s DoH resolution time against AdGuard and NextDNS using dnslookup tool from ameshkov on GitHub.

I also measured network latency with ping.

Cloudflare 1.1.1.1 (no filtering)

dnslookup example.com https://1.1.1.1/dns-query 1.1.1.1

Average resolution time is 40ms.
Average network latency is 5.8ms.

Cloudflare Gateway (with filtering, aka DNS policies)

dnslookup example.com https://<custom>.cloudflare-gateway.com/dns-query 162.159.36.20

Average resolution time is 350ms.
Average network latency is 5.1ms.

AdGuard (public server, with filtering):

dnslookup example.com https://dns.adguard.com/dns-query 94.140.15.15

Average resolution time is 80ms.
Average network latency is 17.7ms.

NextDNS (with filtering)

dnslookup example.com https://dns.nextdns.io/<custom> 193.168.204.73

Average resolution time is 45ms.
Average network latency is 5.7ms.

Measured network latencies do not show an important difference.

However, measured DNS resolution time from Cloudflare Gateway is much higher than the others. I guess the filtering engine is the main bottleneck, but why the resolution time is so much higher than the other DNS providers which are also using a filtering engine.

Do other customers from this community are experiencing the same result?

2 Likes
4

In my testing, comparing https://family.cloudflare-dns.com/dns-query with cloudflare-gateway.com showed different results depending on the Gateway firewall policies configuration, using a tool called resperf to benchmark DNS, and get the average latency in milliseconds (the time between sending the query and receiving a response, excluding connection set-up time, because connections are reused). Pinging family.cloudflare-dns.com and mysubdomain.cloudflare-gateway.com from this system resulted around rtt min/avg/max/mdev = 2.581/2.895/4.246/0.408 ms. With 3 firewall policies active for a location, the latency was 25 ms. With no policies active for a location, the latency was 17 ms. For family.cloudflare-dns.com, the latency was 16 ms. For cloudflare-dns.com, the latency was 9 ms. For dns.adguard.com, the latency was 19 ms.

This was not a very thorough test, I have for example not yet tested uploading a long allowlist or blocklist to Cloudflare Gateway. The difference that I could measure is much smaller than what you measured (perhaps the connection set-up time makes the difference, and I ignored that).

5

Whats going on here?

6

ive noticed the same issue.

7

I can confirm the issue and can provide some additional observations. To confirm the original post, the following behaviour could be observed:

Direct connection (ZeroTrust not active)

dig example.com @1.1.1.1

Average resolution time is 28ms.

ZeroTrust active w/o filtering
(implicitly using Cloudflare Gateways DNS server 192.0.0.2)

dig example.com

Average resolution time is 128ms.

ZeroTrust active with filtering
(implicitly using Cloudflare Gateways DNS server 192.0.0.2)
Filtering 151.000 entries via 151 lists 1.000 entries each. Filtering rules and lists created via Github tool Cloudflare Gateway Pihole Scripts.

dig example.com

Average resolution time is 430ms.

Tested on iOS 17.7.
AS Number: 13335
Cloudflare Data Center: BOM

Interpreting these results suggests a correlation between increased latency and filtering.

However after testing further, this correlation almost fully disappears, when the official WireGuard App (1.0.16 (27)) is used for connecting to ZeroTrust instead of the Cloudflare Warp App (6.26 2409210221.0).

The ZeroTrust WireGuard profile was generated via this method. There is also an easier method available here.

DNS Servers in the WireGuard config were set to the Cloudflare Gateway DNS servers shown in: ZeroTrust Web Interface → Gateway → DNS locations → Default location → IPv4 endpoint & IPv6 endpoint (172.64.XX.1, 172.64.XX.2, 2a06:98c1:54::XXXX:XXXX)

ZeroTrust active via WireGuard App w/o filtering
(implicitly using Cloudflare Gateways DNS servers 172.64.XX.1)

dig example.com

Average resolution time is 38ms.

ZeroTrust active via WireGuard App with filtering
(implicitly using Cloudflare Gateways DNS server 172.64.XX.1)
Filtering 151.000 entries via 151 lists 1.000 entries each.

dig example.com

Average resolution time is 39ms.

Interestingly not only does the additional DNS resolution latency disappear between when filtering is enabled and when filtering is disabled. But also is there no additional DNS resolution latency when ZeroTrust is active or not.

Based on these measurements it seems the issue is likely caused by the Cloudflare Warp App somehow. Maybe there is some issue with how the routing is setup by the app? Or a non-ideal DNS server is used?

8

I was not able to reproduce the issue with the Cloudflare Warp App on MacOS (2024.8.457.0 20240926.11). So it seems to be indeed caused by the Cloudflare Warp App on iOS.

To provide more information on the issue, the Cloudflare Warp App also reports increased DNS resolution times of around 400ms, while the overall network latency is at 29ms:

IMG_5692
IMG_56921170×2532 316 KB

9

IMG_5691
IMG_56911170×2532 191 KB

10

Found a possible workaround/solution:
ZeroTrust support will soon be removed from the Cloudflare Warp app (source). Cloudflare One Agent is the new recommended way to connect to ZeroTrust and Cloudflare Gateway.

With Cloudflare One Agent (v1.6) the DNS resolution latency with extensive DNS filtering enabled (151.000 entries) seems overall much better. It decreased from around 400ms to around 60ms.