Render unauthorizedly overwrote the DNS zones on Cloudflare

[Issue Update] Outage reason:
Render unauthorizedly overwrote the DNS zones on Cloudflare

For Render users, their DNS zones on Cloudflare were unauthorizedly
overwritten by Render at 1am Sep 14, 2021 (UTC). All root domains went
down and were pointed to a Fastly IP unexpectedly.

We use Render to host static sites and Cloudflare as the DNS provider. Thus, Our root domains were hijacked due to the bug. For details, please check Render unauthorizedly overwrote the DNS zones on Cloudflare - #16 by chennien1990 .

[Original Post]

It seems like there was an upgrade of CNAME setup and bulk SSL renewals on Cloudflare yesterday. For the sites activated with CNAME setup, their root domains went down at around 01:00, Sep 14 2021 (UTC); while www and sub-domains were unaffected. As the attached image shown, a strange error message “Not Found” is displayed on every page of the affected domains.

I have several websites activated with CNAME setup via hosting providers’ dashboards. Thus, I tried to make them fix with the following actions, but still not work and realized there might be a bug.

  1. purged caches
  2. re-created the A/AAAA record of root domains
  3. changed root domain’s IPv4/6 address to another servers & different ISPs
  4. deleted the purchased certificates
  5. disabled and enabled universal SSLs
  6. switched to NS setup

Not Found Issue CNAME Setup

Have you opened a ticket? If so, please post the ticket #. support AT cloudflare DOT com

1 Like

Root domains are not technically supported by the CNAME setup.

Only subdomains, not the root domain, can use Cloudflare’s services. This limitation is imposed by Internet DNS specifications.

The “not found” message appears to be coming from the Origin in any case and the Certificate presented by Cloudflare appears to be valid. Is Fastly the origin?

% curl https://startupislandtaiwan.com --dump-header - --silent
HTTP/2 404
via: 1.1 varnish
age: 828
x-served-by: cache-cdg20725-CDG
x-cache: HIT
x-cache-hits: 1
x-timer: S1631723157.844908,VS0,VE0
cf-cache-status: DYNAMIC
server: cloudflare
2 Likes

Ticket no: #2255622 (submitted on Sep 14, 2021). Thank you so much.

To address the issue, a dedicated IP is used on all 3 domains and activated with both NS and CNAME methods. Please note, hearty.tw is now set up via the NS method. However, the root domain is still unreachable with proxy due to it was activated with CNAME method previously.

Our host on Linode is also able to visit via the dedicated IP: https://172.104.68.134 with a SSL warning. You may also add this as an A-record to any Cloudflare site with proxy enabled, it will work without a SSL warning - which proved the Cloudflare’s IPs are blocked by neither the origin server nor the ISP.

The CNAME set-up method on root domains worked for years before Sep 14. It allowed us to create an ALIAS record points to hearty.tw.cdn.cloudflare.net on external name servers; or add an IP address to the name servers manually. I understand there might be a policy change to the method, but the root domains should become resolvable after they are switched back to the NS method. I’m really grateful for your kind assistance. :blush:

Something seems to be very off with the entire configuration I am afraid.

On top of that, based on the provided information, you appear to have an insecure encryption mode selected on Cloudflare.

Hi sandro. Currently, we use full mode of SSL encryption on all cloudflared sites. To make sure the connections are encrypted, domains are also HSTS preloaded. Choosing “full mode” and installing certificates from publicly trusted CAs makes us able to turn off the Cloudflare proxy for websites in emergency. Thank you.

You need Full Strict here. With Full it is still insecure.

Hi sandro. I really appreciate for your suggestion on security. :grinning: I’ll try to upgrade the sites to the strict mode after the issue is solved. Nevertheless, it seems the configuration on SSL mode is not the cause of the outage this time. Because the sub-domains assigned with the same IP address work as expected.

Can you attach a screenshot of the DNS page from the dashboard for one of these domains? Please include the whole page.

When I said the whole page I wanted the two nameservers near the bottom included!

Can you attach a screenshot of the DNS page from the dashboard for one of these domains? Please include the whole page.

When I said the whole page I wanted the two nameservers near the bottom included!

@michael, I really appreciate for your kind assistance. Thank you.

hearty.tw (NS setup)

hearty.me (NS setup)

startupislandtaiwan.com (CNAME setup)

The “not found” message appears to be coming from the Origin in any case and the Certificate presented by Cloudflare appears to be valid. Is Fastly the origin?

There are 6 headers on the not-found page match with Fastly ones. We don’t use Fastly. Probably, for some reason, the records on Cloudflare are overwritten and pointed to Fastly.

Are you using (or have you used) any service provider for your domain, such as WPEngine, Kinsta, Shopify etc. Cloudflare for SaaS takes precedence over your own DNS records, and that looks to me as if the Origin behind hearty.tw is not the same as the origin you are showing in the image above.

The only other alternatives would be a Resolve Override page rule, or a worker on that route which is overriding the origin.

1 Like

Bingo! You’re right! That’s really impressive!

We use Render to create a wildcard redirection on our websites. And Render is actually hosted on Fastly. Our sites work as expected after I removed them on Render.

It’s weird that Render uses Fastly as its CDN and Let’s Encrypt as its SSL provider. How is that possible to overwrite the Cloudflare’s DNS records?

Anyway, @michael, thank you so much for your kind help. :innocent:

Render