Hello Cloudflare Team,
For domains like www.amazon.com or www.netflix.com, 1.1.1.1 is sending ‘uncompressed’ response whereas 8.8.8.8 is sending ‘compressed’ response. Although both are valid responses (adhering to RFC spec), why is there a difference between the two?
Hi, the label compression is heuristic so it doesn’t always match. We’re going to work on improving this in the future.
Thank you, I am not sure I understand what you are saying. The original name server is already sending a ‘compressed’ response (have verified this with ISP DNS servers and 8.8.8.8). Looks like 1.1.1.1 is converting that into ‘uncompressed’ which was a bit surprising as I assumed 1.1.1.1 will only cache what it is receiving from upstream name servers and not override the original ‘compressed’ response to an ‘uncompressed’ one. Can you clarify this behavior please?
My expectation about an implementation that I haven’t personally reviewed is that a DNS Server caches the data in a normalized internal format, not specific raw queries, and answers queries from the internal data.
There are a number of cases where the server will need to return slightly different answers to similar (but not identical) queries (for example, returning a cached A record that is useful when a query to a CNAME targets that label).
Certainly this is an implementation choice.
Hi Dave, I wanted to bring the following to your attention. Since Cloudflare is sending uncompressed DNS responses, some IPTV settopboxes of the Dutch ISP KPN are failing to connect to the ISP network for television service.
The reason for this is that these IPTV STBs do a TXT lookup in DNS to find the update multicast group.
Many people use a Ubiquity USG as Internet gateway, which by default uses Cloudflare DNS (1.1.1.1).
When the Cloudflare DNS returns an uncompressed response on the TXT lookup, the STB cannot process the DNS record, causing a boot loop and no service.
This problem can be solved in three ways:
- people will use their ISP supplied Internet Gateway (defeats the purpose of BYOD gateway)
- ISP updates STB code to be capable to process uncompressed DNS responses (ISP updates takes ages)
- The Cloudflare DNS starts returning compressed DNS responses.
Although the RFC does not forbid the return of uncompressed DNS, compressed DNS is more common on the Internet now. Cloudflare is the only public resolver that returns an uncompressed RR.
Test with: dig -t TXT sap.stb.itvonline.nl @SERVER|grep “MSG SIZE”
-
OpenDNS
208.67.222.222 → ;; MSG SIZE rcvd: 82
208.67.220.220 → ;; MSG SIZE rcvd: 82 -
Cloudflare
1.1.1.1 → ;; MSG SIZE rcvd: 102 ← larger because not compressed
1.0.0.1 → ;; MSG SIZE rcvd: 102 ← larger because not compressed -
Google Public DNS
8.8.8.8 → ;; MSG SIZE rcvd: 82
8.8.4.4 → ;; MSG SIZE rcvd: 82 -
Comodo Secure DNS
8.26.56.26 → ;; MSG SIZE rcvd: 82
8.20.247.20 → ;; MSG SIZE rcvd: 82 -
Quad9
9.9.9.9 → ;; MSG SIZE rcvd: 82
149.112.112.112 → ;; MSG SIZE rcvd: 82 -
Verisign DNS
64.6.64.6 → ;; MSG SIZE rcvd: 82
64.6.65.6 → ;; MSG SIZE rcvd: 82
The full troubleshooting thread can be found here in Dutch: [Ubiquiti & IPTV] Ervaringen & Discussie - Netwerken - GoT
This is fair.
Ultimately I would suggest the problem is these devices have an incomplete DNS implementation which relies on optional features. Soapbox rant: A lot of the internet is broken because implementing protocols is done by guessing at what seems to work rather than implementing a specification properly. This is especially bad because there are so many libraries that “just work”, and having different implementations that follow the same specifications but do not attempt to reproduce byte-identical responses is ultimately healthy as it helps identify bad/broken code.
Another example that you won’t see until you set up a not-identical-to-every-consumer network, a lot of IOT gear (even including networking gear) struggles with any IPv4 subnet size other than a /24. I’ve run into more than one piece of gear where a 192.168.0.0/22 subnet, with a default gateway of 192.168.0.1, treats devices assigned 192.168.1.1 specially. My ISP modem won’t let me include 1.1 in a DHCP range because it conflicts with the default gateway, which it clearly does not, but the UI developers failed to understand how subnets work and just chop the last octet and compare it. A previous modem excluded all .1 addresses from the internal firewall (assuming the traffic would be from the gateway itself).
Monoculture, at both the implementation and configuration levels, is bad.
Nonetheless, it would be nice to see Cloudflare step up to the modern state of technology here as this seems like a feature that should exist.
1 Like
1.1.1.1 does support label compression, but in this case the problem is the compression starts from the first name in the answer section, not from the question. We’re working on it, but it’s a bit longer process than expected.
2 Likes
Hi,
It’s been a while since the last post on this topic. Are there any updates?