All my Production Loadbalancers dissapeared at 10am PST

Website, Application, Performance
1

What is the name of the domain?

voccent.com

What is the error number?

111

What is the issue you’re encountering

All traffic LB disaapeared

What steps have you taken to resolve the issue?

2

There is no info for anyone to go on here so I’ll just give the info I can gather.

You have no DNS records for your apex:

$ dig voccent.com

; <<>> DiG 9.10.6 <<>> voccent.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13483
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;voccent.com.			IN	A

;; AUTHORITY SECTION:
voccent.com.		1545	IN	SOA	adam.ns.cloudflare.com. dns.cloudflare.com. 2352526989 10000 2400 604800 1800

;; Query time: 2 msec
;; SERVER: 192.168.178.1#53(192.168.178.1)
;; WHEN: Sun Sep 22 22:49:31 BST 2024
;; MSG SIZE  rcvd: 116

But you do for www

$ dig +short www.voccent.com
104.26.12.125
104.26.13.125
172.67.72.161

Which works - https://www.voccent.com/

So, I’d recommend creating a CNAME from apex → www

Any more info at all here? What does your LB config say? LB analytics? Audit log? There is no information for anyone here.

3

Thank you, but 4h back it was working with the config we didn’t change for 4 years.
All our LB was removed. Even though we pay for the account.
dig you tried above is what I’m trying to recover right now by memory.

app.voccent.com is missing too.

4

When I’m trying to add it back, they have 500 errors on backend in the browser. But [email protected] doesn’t respond. Also, a reminder:

  1. paid account
  2. no changes for 4 years
  3. audit shows that last actions on Sep 22,2024 was user:cloudflare removed resources.
5

So sounds like they were removed - did you have any unpaid invoices? Nothing should be automatically removed unless there wasn’t a payment for some time

When was the last invoice for your LBs issued?

6

I’m recreating everything. Using UI, this is crazy.
We need to have more alternatives than Cloudflare. Not surprised their stocks go down overtime.

7

All paid. Actually ahead of time as usual.
Last invoice - few days back. Also, if they decided to shut down something, an email would come, we didn’t get any, except for the email that services are down just 4h back.
Also, argo was paid recently and it was down too.

8

It sounded like you have a support ticket, what’s the ticket number?
I’ll escalate this for support to take a look

9

@WalshyMVP I really appreciate for your help, really. Usually tickets stuck and community is dead, but you are like a light at the end of the tunnel! Thank you.
I do not have a ticket, we don’t pay that level of tier. We’re on Pro.

10

Alright, no worries, let me escalate and they can make a ticket for you.

1 Like
11

I recreated basic functionality to bring up a service (Africa is waking up soon), other things our SRE will start woking on in 6h.

Again, @WalshyMVP thank you for your help.

Steps we performed to fix:

  1. Delete LB
  2. Wait 10min
  3. Create a new LB
  4. Recreate HealthChecks
  5. Wait
  6. Up online

Scares:
What if tomorrow we will get the same problem again.

1 Like
12

Thank you a lot!

13

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.