The DoH TXT record escape has been changed

What is the name of the domain?

endpoints.adm.allxon.com

What is the issue you’re encountering

When use DoH to get dns record, the response content is differ from before

What steps have you taken to resolve the issue?

We’ve configured a DNS TXT record for endpoints.adm.allxon.com to provide API endpoint information.
The record’s data is a JSON string, originally formatted as follows:

"api-v1={\"region\":\"GLOBAL\",\"endpoints\":{\"API\":[\"api.adm-dev.allxon.com\"]}}"

We use Cloudflare’s DoH, https://cloudflare-dns.com/dns-query?name=endpoints.adm.allxon.com&type=TXT with the Accept: application/dns-json header, to retrieve this information.
This has been working reliably, but we’ve recently encountered a change in the response format that’s impacting our service.

Here’s the current response from Cloudflare’s DoH service:

{
    "Status": 0,
    "TC": false,
    "RD": true,
    "RA": true,
    "AD": true,
    "CD": false,
    "Question": [
        {
            "name": "endpoints.adm.allxon.com",
            "type": 16
        }
    ],
    "Answer": [
        {
            "name": "endpoints.adm.allxon.com",
            "type": 16,
            "TTL": 300,
            "data": "\"api-v1={\\\"region\\\":\\\"GLOBAL\\\",\\\"endpoints\\\":{\\\"API\\\":[\\\"api.adm.allxon.com\\\"]}}\""
        }
    ]
}

Comparing this to a previous, correct response from our logs:

{
    "Status": 0,
    "TC": false,
    "RD": true,
    "RA": true,
    "AD": true,
    "CD": false,
    "Question": [
        {
            "name": "endpoints.adm.allxon.com",
            "type": 16
        }
    ],
    "Answer": [
        {
            "name": "endpoints.adm.allxon.com",
            "type": 16,
            "TTL": 300,
            "data": "\"api-v1={\"region\":\"GLOBAL\",\"endpoints\":{\"API\":[\"api.adm-dev.allxon.com\"]}}\""
        }
    ]
}

We’ve also tested with Google’s DoH (dns.google):

{
    "Status": 0,
    "TC": false,
    "RD": true,
    "RA": true,
    "AD": true,
    "CD": false,
    "Question": [
        {
            "name": "endpoints.adm.allxon.com.",
            "type": 16
        }
    ],
    "Answer": [
        {
            "name": "endpoints.adm.allxon.com.",
            "type": 16,
            "TTL": 300,
            "data": "api-v1={\"region\":\"GLOBAL\",\"endpoints\":{\"API\":[\"api.adm.allxon.com\"]}}"
        }
    ],
    "Comment": "Response from 205.251.198.198."
}

The key difference is in the escaping of the JSON string within the TXT record’s data field.
Previously, backslashes (\) were used to escape the inner double quotes within the JSON.
Now, Cloudflare’s DNS service is returning the JSON with triple backslashes (\\\).

This change breaks our service. Our code expects the original escaping and is unable to correctly parse the JSON when it contains the triple backslashes.
Google’s DNS, as you can see, returns the original, correctly escaped JSON.

Could you please investigate this change and, if possible, revert to the previous escaping mechanism (single backslashes)?
This would greatly help restore our service functionality.
We understand that changes happen, and we appreciate your prompt attention to this issue.
Thank you for your assistance!

What feature, service or problem is this related to?

DNS records

2 Likes

Hi there!

I believe the new behavior is correct. TXT records consist of “strings” surrounded by quotes. In your example, these appear as \" due to JSON escaping. The previous response shows the quotes starting/ending the string the same as literal quotes inside the string, which can’t be right.

Google’s API unfortunately does not show quotes around strings at all, simply concatenating the strings into one and then returning that as-is. This means you lose some information, but I don’t think there’s a standard for these APIs, so they’re free to do that. That explains why there’s one less level of escaping needed though.

If you use jq -r (or any JSON parser) to undo the JSON-level escaping, you should see the same output as you’d receive from DiG, for example:

$ curl -sH 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=endpoints.adm.allxon.com&type=TXT' | jq -r '.Answer[0].data'
"api-v1={\"region\":\"EU\",\"endpoints\":{\"API\":[\"api-eu.adm.allxon.com\"]}}"
$ dig +short endpoints.adm.allxon.com TXT | grep EU                                                                                           
"api-v1={\"region\":\"EU\",\"endpoints\":{\"API\":[\"api-eu.adm.allxon.com\"]}}"

Alternative explanation, decoding step by step:

# JSON
$ curl -sH 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=endpoints.adm.allxon.com&type=TXT' | jq '.Answer[0].data' 
"\"api-v1={\\\"region\\\":\\\"EU\\\",\\\"endpoints\\\":{\\\"API\\\":[\\\"api-eu.adm.allxon.com\\\"]}}\""

# actual string within JSON
$ curl -sH 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=endpoints.adm.allxon.com&type=TXT' | jq -r '.Answer[0].data'
"api-v1={\"region\":\"EU\",\"endpoints\":{\"API\":[\"api-eu.adm.allxon.com\"]}}"

# actual string within the DNS TXT record, within JSON
$ curl -sH 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=endpoints.adm.allxon.com&type=TXT' | jq -r '.Answer[0].data' | jq -r
api-v1={"region":"EU","endpoints":{"API":["api-eu.adm.allxon.com"]}}

(Using jq -r again in the last command isn’t strictly correct because DNS escaping rules aren’t the same as JSON escaping rules, but it works well enough for purposes of this example.)

I’ll pass this thread along to the Resolver team though in case they have anything to add.

5 Likes

Hi Cloudflare Team,

Thanks for the detailed explanation.
We understand that the current escaping behavior might be technically correct according to the standards, and we appreciate you clarifying the different levels of escaping.

However, our service has relied on the previous behavior for years.
Our service runs on edge devices that have already been deployed to our customers.
These devices have a backup mechanism: if they can’t reach Cloudflare’s DoH service (cloudflare-dns.com), they fall back to a direct DNS query.

Because of this change, to work around this right now, we’re unfortunately having to ask our customers to block cloudflare-dns.com to force the fallback to direct DNS. This is obviously not an ideal solution, and we’re concerned about the long-term implications.

We understand that changes are sometimes necessary, but this change is effectively a breaking change for us, and we weren’t prepared for it. It requires us to update the software on all of our deployed edge devices, which is a significant undertaking.

Therefore, we have two requests:

Non-Breaking Change (Ideal): Could Cloudflare consider providing this new escaping behavior as part of a versioned API (e.g., a v2 API) so that we can migrate at our own pace? This would be the most ideal solution and would prevent disruption to existing services.

Grace Period (Minimum): If a versioned API isn’t feasible, could the Cloudflare team provide a grace period of at least a couple of months before enforcing this new escaping behavior? This would give us the necessary time to update the software on all of our deployed edge devices and avoid the current workaround of blocking cloudflare-dns.com.

We understand the complexities involved in managing a service like Cloudflare DNS, and we appreciate your understanding and willingness to help us find a solution that minimizes disruption to our users. We believe a grace period or a versioned API would be a reasonable and helpful compromise.
Thanks again for your time and consideration.

Wonder if it’s related to the recent post here :thinking:

Yes, we found a related post without a recent reply. I’ll reply to it with this post.

Apologies for any inconvenience this caused. We have decided to start rolling back this change until at least March 31 2025. Starting April 2025 we will re-release this change, as it closer aligns with the expected behavior of TXT records.

As the DoH JSON format is not standardized, we’d recommend you to move to the DoH wireformat to prevent changes like this from causing issues in the future.

5 Likes

Thank you for the prompt response and for deciding to roll back the TXT record escaping change. This is excellent news and will greatly alleviate the disruption to our service. We really appreciate you providing the grace period until March 31, 2025.

We’ve noticed that the response format has been rolled back to the original format in many cases, but we’re still occasionally seeing responses with the triple backslashes. We wanted to bring this to your attention.

We have one main question:

While we are working on migrating to the change, we might need more time beyond March 31, 2025. What recommendations do you have for us in that scenario? Would it be possible to discuss extending the grace period on a case-by-case basis if needed? We would also be interested to know if there is a way to ensure that only our domain’s query results are kept in the original format, even if that requires a paid service or dedicated configuration. This would be a valuable option for us.

Thank you again for your understanding and assistance.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.