DNS Flooding? Could changing name servers in a certain time span invalidate the name servers?

We just had a massive outage that lasted for 2 days.

TL;DR: We made a recent update to our nameservers in which we fixed a typo in one of CloudFlare’s NS records in our registrar and apparently that brought the entire system down. DNS propagation didn’t occur (it was just partially done to several places) Why was this?

Story:

In our company, we use several third-party providers to provide IT needs. We acquired a Barracuda Email Total Protection license that needed to be installed on our Office 365 space. When we were about to configure it, we discovered that one of the third-party providers had rerouted the traffic through Cloudflare, and Office 365 needed to add the special MX records and needed to set them into the Cloudflare account.

The problem was that the third-party provider got indicted and put into jail after being related to a corruption case in our country (Not the US), and we lost access to the Cloudflare credentials (he wasn’t willing to give them back).

This was problematic as we didn’t know all the endpoints that the Cloudflare distribution was pointing to (Our domain routed VPNs, Website, and who knows what), so simply rerouting Cloudflare wasn’t a simple solution.

Lo behold the changes:
That’s when I started to make some personal changes to the registrar (Only namespaces, nothing else was altered). I first added the original name servers from the registrar in addition to Cloudflare’s to see if the email configuration needed by Office 365 could proceed… in the end, it couldn’t.

I then tried removing Cloudflare’s and leaving the registrar’s default ones, but that brought down the VPN access almost instantaneously. That’s when we reverted to Cloudflare’s.

But what happened, about 3 weeks passed and we noticed that there were some VPN issues (2 days ago, around Monday)… The problem apparently was a misspelling in one of Cloudflare’s original name servers.

Instead of barbara.ns.cloudflare.com we mistakenly typed barabara.ns.cloudflare.com (Note the redundant “a”).

We proceeded to fix the typo thinking it was a simple solution… and then BOOM! The name servers didn’t update properly, and the entire domain was down. We thought it was a propagation issue and waited for almost 48 hours before taking additional action.

After seeing it was taking too long, we decided to contact another third-party provider (The one who hosts our email and website) which ended up making some configurations to the registrar’s name servers (purging Cloudflare in the process), and gaining access to the emails, and website.

As I’m typing, we’re trying to fix the VPN issues so we can get back online.

The third-party provider which fixed the issue told us we caused a DNS flooding and that triggered Cloudflare to somewhat blacklist us. I don’t think that’s exactly what happened, but wanted to dig in to learn more.

As with every IT catastrophe, it’s paramount to learn from the mistakes. We wanted to know what caused the issue, and what can we do to avoid it in the future.

Thanks!!

Sounds like someone failed to copy and paste the name servers when setting it at the registrar. Going forward use copy/ past and have someone doublecheck the spelling before hitting submit.

This will almost definitely not do what you intend, but will cause a lot of confusion. I understand the desire here to get Microsoft 365’s validation record through though, this is a tough place to be when you don’t have the full configuration available to you.

This makes sense. Unless your registrar’s DNS is complete and covers all your configuration and you have a replacement for any other services that Cloudflare was providing, things will break.

Okay, so I am thinking that maybe Cloudflare noticed the nameservers didn’t match, and started sending alerts to the account owner, which were (obviously) ignored. Things would (mostly) work in the meantime with one correct NS in terms of DNS resolution, but Cloudflare would still eventually disable the site because the nameservers don’t match.

1 Like

Daaaaaaannggggg. It makes sense :man_facepalming:. Thank you very much!!!

Well… I totally agree with you. It was entirely my fault. I couldn’t copy and paste it as I was giving remote assistant via a Microsoft Teams’ video call.

As far as avoiding in the future, if you sign up for Cloudflare yourself, you can then grant a consultant or employee access under their own credentials, but you still retain ownership.

2 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.