Disabled CNAME flattening, cloudflare still responds with cached records

joost2 · June 24, 2022, 7:42am

Hi,

I have a CNAME record in Cloudflare DNS as follows, to delegate acme DNS requests to an external DNS:


CNAME     _acme-challenge     MYDOMAIN.com.acme.nhcloud.MYDOMAIN.com     [DNS only] TTL 1 min


This is used for LetsEncrypt DNS acme validation. This has started to fail recently. I figured this might be due to CNAME flattening, which I had enabled for all hosts in the zone. I've now switch to 'Flatten CNAME at root'.

Yet, more than 8 hours after disabling CNAME flattening, cloudflare DNS is still responding with cached TXT records:


dig _acme-challenge.MYDOMAIN.com txt @nile.ns.cloudflare.com

; <<>> DiG 9.16.1-Ubuntu <<>> _acme-challenge.MYDOMAIN.com txt @nile.ns.cloudflare.com

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42793

;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 1232

;; QUESTION SECTION:

;_acme-challenge.MYDOMAIN.com. IN TXT

;; ANSWER SECTION:

_acme-challenge.MYDOMAIN.com. 300 IN TXT "gBj9M_7HM (redacted) 1kp_wF8SJHaAYFk2GAsSW60UM"

_acme-challenge.MYDOMAIN.com. 300 IN TXT "fCkX6SLIL (redacted) PkS3FxaGxuJFsk"

_acme-challenge.MYDOMAIN.com. 300 IN TXT "r3fu7dceO (redacted) rtO5_tbB_seawtAwnCsNeX1c"

_acme-challenge.MYDOMAIN.com. 300 IN TXT "Fcr1aYboor (redacted) H-g4J1YYmVCYtVvvto"

;; Query time: 8 msec

;; SERVER: 2a06:98c1:50::ac40:21d6#53(2a06:98c1:50::ac40:21d6)

;; WHEN: Fri Jun 24 09:18:47 CEST 2022

;; MSG SIZE rcvd: 278


The other weird thing is I'm seeing multiple TXT records, while my actual DNS (acme-dns-certbot) is only responding with a single one. It seems your CNAME flattening is eternally caching my TXT records.

I've tried renaming and deleting the _acme-challenge entry, but as soon as I add it again, I'm served the cached data once again. So I'm stuck and can't renew my letsencrypt certificate.

Also I would expect this to work, even with CNAME flattening enabled. The certbot DNS is responding with 60 seconds TTL.

This looks like a bug. Could someone contact me to resolve this issue?

Thanks from a Cloudflare Pro customer

joost2 · June 24, 2022, 7:43am

and sorry for the weird formatting, for some reason your forum is replacing every occurrence of the word CNAME with a link, and then erroring because I have more than 4 hyperlinks in my post

KianNH · June 24, 2022, 9:44am

CNAME flattening has nothing to do with caching records - CNAME flattening turns CNAMEs into IP addresses.

What are you saying is ‘cached’? If you have a CNAME and lookup the TXT records for that CNAME, you will get the TXT records from the target your CNAME is pointing to.

joost2 · June 24, 2022, 11:39am

What are you saying is ‘cached’? If you have a CNME and lookup the TXT records for that CNME, you will get the TXT records from the target your CN*ME is pointing to.

Indeed, that’s what I expect. But cloudflare is still returning TXT records.

$ dig _acme-challenge.MYDOMAIN.com txt @nile.ns.cloudflare.com

; <<>> DiG 9.18.1-1ubuntu1.1-Ubuntu <<>> _acme-challenge.MYDOMAIN.com txt @nile.ns.cloudflare.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11642
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.MYDOMAIN.com.	IN	TXT

;; ANSWER SECTION:
_acme-challenge.MYDOMAIN.com. 300	IN	TXT	"gBj9M_HM1JRlQ (redacted) JHaAGAsSW60UM"
_acme-challenge.MYDOMAIN.com. 300	IN	TXT	"fCkX6SLILF2x_XU (redacted) BQBCNxuJFsk"
_acme-challenge.MYDOMAIN.com. 300	IN	TXT	"r3fu7dceO (redacted) tbB_se1yQqtAwnCsNeX1c"
_acme-challenge.MYDOMAIN.com. 300	IN	TXT	"Fcr1aYboorH (redacted) 1YYmVCYtVvvto"

;; Query time: 12 msec
;; SERVER: 173.245.59.214#53(nile.ns.cloudflare.com) (UDP)
;; WHEN: Fri Jun 24 13:32:06 CEST 2022
;; MSG SIZE  rcvd: 278

$ dig _acme-challenge.MYDOMAIN.com cname @nile.ns.cloudflare.com

; <<>> DiG 9.18.1-1ubuntu1.1-Ubuntu <<>> _acme-challenge.MYDOMAIN.com cname @nile.ns.cloudflare.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38991
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.MYDOMAIN.com.	IN	CNAME

;; ANSWER SECTION:
_acme-challenge.MYDOMAIN.com. 60	IN	CNAME	MYDOMAIN.com.acme.nhcloud.MYDOMAIN.com.

;; Query time: 12 msec
;; SERVER: 172.64.33.214#53(nile.ns.cloudflare.com) (UDP)
;; WHEN: Fri Jun 24 13:32:15 CEST 2022
;; MSG SIZE  rcvd: 91

$

Letsencrypt is asking for TXT records and gets those stale responses directly from cloudflare. Cloudflare should instead return the CN*ME response.

I don’t want to post my domain name here in a public forum, can someone from CF contact me?

KianNH · June 24, 2022, 11:40am

I can’t really offer any other advice without knowing the domain.

Open a support ticket.

joost2 · June 24, 2022, 11:45am

How do I open a support ticket?

‘Technical Problem’ just points me here.

joost2 · June 24, 2022, 11:55am

Apparently I can email [email protected]
That was well hidden.

KianNH · June 24, 2022, 11:58am

Out of curiousity, have you tried running the dig command with a recursive server like 1.1.1.1?

dig _acme-challenge.domain.com TXT @1.1.1.1

Yours are asking nile.ns.cloudflare.com which is an authoriative nameserver, not a recursive one - I’m not entirely sure if that’ll ever follow the CNAME chain, I’d assume you’re just asking Cloudflare for their TXT records.

joost2 · June 24, 2022, 12:11pm

dig _acme-challenge.domain.com TXT @1.1.1.1

This returns the same (faulty) TXT records (otherwise I assume letsencrypt would work as well).

As I said I had CNAME flattening enabled until yesterday (24h ago). Those TXT records must be at least 1 week old (and they have a TTL of 1 minute).

joost2 · June 24, 2022, 12:13pm

Also I would expect CNAME flattening to work transparently and always return fresh data (after TTL expired)

KianNH · June 24, 2022, 12:23pm

It does - but I don’t think TXT records are within the realm of CNAME flattening.

CNAME flattening will turn your CNAME into IP addresses - if it also populates the other record types such as TXT from the original CNAME isn’t something that I’m aware of and also isn’t documented.

If you’ve already contacted support then they should be able to give you a definitive answer.

sandro · June 24, 2022, 12:28pm

Of course not, what should it flatten

KianNH · June 24, 2022, 12:30pm

I mean in the context of if you had a CNAME for foo going to foo.example.com then a TXT lookup for foo would return the TXT records from foo.example.com

With CNAME flattening, I don’t know if the TXT lookup for foo returns the ones on Cloudflare (since it’ll accept records alongside a CNAME) or the ones from foo.example.com.

sandro · June 24, 2022, 12:31pm

Sure, of course, that’s what DNS does. I think there’s some confusion what flattening is

A TXT record is a TXT record and resolving it will always return its content.

joost2 · June 24, 2022, 12:31pm

It could proxy the TXT records. There’s not much benefit, but why not? And apparently that’s what they are doing, otherwise I don’t see where those TXT results are coming from.

I’ve opened a support ticket, will let you know.

sandro · June 24, 2022, 12:32pm

You can’t proxy TXT records.

Equally, your support ticket is unlikely to get an answer, unless you are on a paid plan.

Anyhow, @KianNH’s response above is spot on.

joost2 · June 24, 2022, 12:35pm

That’s the issue, I don’t want my TXT records to be proxied. And I am on the Pro plan.

KianNH · June 24, 2022, 12:35pm

No CNAME flattening:

;; QUESTION SECTION:
;protonmail._domainkey.kian.org.uk. IN	TXT

;; ANSWER SECTION:
protonmail._domainkey.kian.org.uk. 300 IN CNAME	protonmail.domainkey.d765ws4plgnyzhuqqyqmnohwl4jfhs6fpp6rirnozmlua5dytl22q.domains.proton.ch.
protonmail.domainkey.d765ws4plgnyzhuqqyqmnohwl4jfhs6fpp6rirnozmlua5dytl22q.domains.proton.ch. 0	IN TXT "v=DKIM1;k=rsa;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu8hHZ2FVBjQcGuDg7BlCcZ/c7CqICnwEiq5QFmSxO/AuXJuZWHuUcuKVXWlcuNkvGd0cAo7mS84riG1D/3k86eus4h8UC13Kk5OAkveGZyUhxgtAgpX6LjoLy88rieTUztLqLXAqyIMh4L6uflCP9S/tfHu+rr6+JqsD14Z59ggfydSPJf0sL8O+Sdj7g9dRfYI" "cvJKTnBGqlIvZyTYOtEAw4P2lPS+RY7cvgbUMobgeldCGSUcfgt2OCvEw+WJUoE8sN0wO/H+F1VVybfAFhz8fB2lq3Yf07v2ydBXhqYo0OZPqTlwFCB9Q817bgMAgtQ6iAb84rMxksn2Q1IzyUQIDAQAB;"

The lookup follows the CNAME chain to find the TXT records on protonmail.domainkey.d765ws4plgnyzhuqqyqmnohwl4jfhs6fpp6rirnozmlua5dytl22q.domains.proton.ch.

With CNAME flattening:

;; QUESTION SECTION:
;protonmail._domainkey.kian.org.uk. IN	TXT

;; ANSWER SECTION:
protonmail._domainkey.kian.org.uk. 264 IN TXT	"v=DKIM1;k=rsa;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu8hHZ2FVBjQcGuDg7BlCcZ/c7CqICnwEiq5QFmSxO/AuXJuZWHuUcuKVXWlcuNkvGd0cAo7mS84riG1D/3k86eus4h8UC13Kk5OAkveGZyUhxgtAgpX6LjoLy88rieTUztLqLXAqyIMh4L6uflCP9S/tfHu+rr6+JqsD14Z59ggfydSPJf0sL8O+Sdj7g9dRfYI" "cvJKTnBGqlIvZyTYOtEAw4P2lPS+RY7cvgbUMobgeldCGSUcfgt2OCvEw+WJUoE8sN0wO/H+F1VVybfAFhz8fB2lq3Yf07v2ydBXhqYo0OZPqTlwFCB9Q817bgMAgtQ6iAb84rMxksn2Q1IzyUQIDAQAB;

Cloudflare does carry over other record types which was what I was wondering about.

So CNAME flattening or not, the TXT records of your CNAME target would be returned.

sandro · June 24, 2022, 12:35pm

And your TXT won’t be proxied.

All right, on a Pro plan you should get a response, but @KianNH already addressed it, so you can close the ticket.

joost2 · June 24, 2022, 12:37pm

With all respect @sandro, but my problem is not solved until letsencrypt sees the correct TXT records.

Topic		Replies	Views
Unproxied CNAMEs Refusing to resolve DNS & Network	3	15	January 15, 2025
My cloudflare account is not picking up the CNAME _acme-challenge for subdomain DNS & Network CJ24N	2	377	May 17, 2024
CNAME Flattening as DNS Record Type DNS & Network	12	5924	June 13, 2021
DNS Server Maintenance - CNAME Record on Cloudflare DNS & Network	4	2307	June 6, 2020
ACME_CHALLENGE not working DNS & Network	2	351	April 17, 2024

Disabled CNAME flattening, cloudflare still responds with cached records

Related topics