All my sites down with 503 Error caused by Cloudflare

Someone changed ALL “A - Records” in my domains which caused a 6 to 9 hour shutdown of ALL these sites, until I (along with my hosting support), got them all “reverted” to what they SHOULD be. Is it possible that Cloudflare restored a backup of certain information, which caused all A - records to be overwritten?

Cloudflare does not change your records.

If you believe your records were changed, check your Audit Log:
https://dash.cloudflare.com/?to=/:account/audit-log

2 Likes

Thank you @Laudian , I checked the log as you suggested. It shows a bunch of things that my own actions MAY have caused, but I couldn’t know for sure, as I was not changing certificates and things like that.

Below is a complete explanation of what was going on, and why:

In late November, my hosting provider for all these sites did a MAJOR server upgrade, and therefore prompted me to change ALL “A” and “AAAA” records to the new IP address. About 10 days later, (yesterday), almost ALL of those records had been changed to some other IP, and nearly all of my sites [DNS and cache serviced by] Cloudflare showed a “503” error on any browser, and on UpTimeRobot.

I spent several hours last night in a chat session with the hosting provider, and we eventually determined the changes occurred above. I did NOT make the changes that caused the error, so someone at Cloudflare DID make them. My account has 2FA on it, so not likely hacked, and I don’t have anyone else [helpers] who has access other than myself.

Hmmmm…

If the records were changed, the Audit Log would explicitly tell you that the records were changed. Do you see any such events (type update, resource dns.record) in your logs?

Are the sites working now that you changed the records again?

Cloudflare does not ever, under any circumstances, make DNS changes on their customers behalf.

2 Likes

Thank you, I heard you say that the first time.

I will change my password then, hoping this will not cause a cascade of problems across my sites. I think all the intertwining (CF plugin, namely) on the sites is by API only.

As far as audit log goes, we are talking about 14 sites, so potentially 42 log entries that were created FOR THAT PURPOSE ALONE over the past 15 days or so. I will have to pore over it some more to understand thoroughly what occurred.

I SAY AGAIN, I did not make the changes that took my sites down. So, I will change the [already cryptic-random] password.

Yes, after my several hours of effort last night, all the sites are once again functional.

1 Like

I don’t believe you should change your password, especially as you have 2FA activated.

A 503 would also be very untypical for a hacked account.

But you should definitely go through the log. If the changes were made yesterday, there shouldn’t be more than ~10-30 entries for that date, and checking those shouldn’t take you more than a few seconds each.

You are really interested in Rec del, Rec add and Update.

Many people experience reverting DNS records when they use external management tools like Ezoic. The changes may also be made with an API Key or Token, which would not be subject to 2FA.

1 Like

Thank you, I think you are right about NO need for passw change… All of this came from internal CF source as far as I can see. Interestingly, the ONLY IP address changes made on A-records were from MY OWN username… ONLY!

Your last comment is helpful. HOWEVER… the “user” who made some changes was “Cloudflare”, (as opposed to my own username) and it was apparently by API; it was concerning the last of several “Rec del” or “Rec add” actions interspersed with other things, like certificate changes, etc.

The “content”: ""g1h139VUj … of the OLD VALUE / NEW VALUE was always encrypted, as shown here. No VISIBLE IP addresses. Again, the surrounding actions by “Cloudflare” seems to been having to do with SSL certificate changes. But still, somehow they managed to change all the A records of EVERY DOMAIN and sent them into oblivion.

I have pasted below a PORTION one of the record changes so as not to divulge private info.
---------------snip--------------
Date:
2023-12-06T03:45:54-08:00

Resource:

DNS_record

Interface:

API

Audit Record:

3b4539----------------------------------

Metadata:
{
“grpc_client_name”: “bushbaby”,
“zone_name”: “thechurchofwhatshappeningnow_com”
}

The record you’ve shown is an acme-challenge record. This type of record is used to provision SSL certificates. The content is not “encrypted”, but just a random string.

There is no visible IP address because this is a TXT record, not an A/AAAA record, so there is no IP, just text.

Bushbaby is the name Cloudflare uses for certificate provisioning.

December 6th was not yesterday. You should only look at the logs from yesterday.

1 Like

Thank you for the “translation” – LOL…

Yes, I’m very aware it was from an earlier date, as I was looking into the past to find something that I DIDN’T DO. The logs from yesterday/day before only show my own actions, which were to correct the IP addresses in the A-records involved.

I will keep a good eye on UpTimeRobot for anomalies across the sites that it monitors.

Thank you for all your time here… hopefully everyone learned something! :slight_smile:

:slightly_smiling_face:

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.