Under an origin server DDoS - How are attackers finding my origin IP?

Hello,
For the last few weeks my origin server has been under a sustained 1GB per second DDoS attack. Some of you might have read about it here. ( I know Cloudflare can’t help me if my origin is exposed, bear with me )
After working with my hosting provider to mitigate some of the traffic, changing IP addresses, verifying firewall rules and finally moving to an entirely new data center, the attacks have continued with a very short turnaround. After an IP address change, the DDoS attack would resume in less than 2 hours.

My main question is how are they doing this? How are they able to bypass Cloudflare and discover my origin IP so effortlessly? I’ve read the Cloudflare documentation on best practice, and I think I’m doing everything right. I have to be missing something painfully obvious, and at this point, I’m out of ideas about what I can do here. It seems no matter the mitigation I try, the attackers seem to have unlimited resources.

Below is a summary of what I’m working with, the origin server is running Ubuntu Linux 20.4 LTS:

  • Origin server configured to only accept TCP traffic on port 443 from the Cloudflare IP address list.
  • SSH port configured using key based authentication, and only accepts connections from a specific IP address.
  • Server IP isn’t listed on Shodan.
  • Server IP isn’t listed on Censys.
  • Cloudflare proxying A and AAAA records.
  • No mail servers.
  • No rDNS record set.
  • Hosting provider firewall enabled, also whitelisting Cloudflare IP addresses.
  • No rate limiting for Cloudflare IP addresses.
  • Some TXT records are present for Google Verification & KeyBase verification.

During the attack, my origin NIC is flooded with UDP packets, it looks to me like just junk packets to try and drain system resources. Errors seen on the frontend are a combination of 520, 521, 522, 524 and 526, with POST requests being the slowest.

I should mention I am a Cloudflare Free plan member.

Can anyone provide some insights into what could be happening here? I know there are a multitude of ways to discover origin IP addresses, but this is getting out of hand for me and I don’t know what I’m missing. Please feel free to ask for any configuration clarifications.

Thank you.

I’ll keep it short and sweet:

  1. Bummer
  2. Maybe they’re just good detectives…or lucky. It really doesn’t matter or change the outcome.
  3. See if your host can use an upstream firewall.
  4. Use Argo Tunnel so you can drop all inbound traffic.

p.s. UDP? Why is that even accepting packets?

I agree with you, it is an absolute bummer.
Argo tunnels would help me here, but the sheer volume of data I push each month makes the cost just too much. I’ve seen people suggest trycloudflare, but I don’t think that’s what it’s designed for.

Edit: What’s an upstream firewall?

Upstream firewall in their network hardware. It’s a hassle if the host isn’t already set up for it.

Argo costs? I swear…(not at you)…we @MVP keep thinking you don’t need an Argo subscription to do this and nobody’s come along to prove us wrong.

I am afraid no, as mentioned in the other thread, they’d still saturate your network. Argo will be useless here.

If I kept the same origin IP, yes. If I got it changed and instantly setup Argo, would that not work?

No, because that’s what you have done and they still saturate your network.

The current problem is they’re saturating the NIC. Moving the problem upstream should relieve local saturation. Though their host might not like all that unwanted attack traffic. But they sound accommodating so far.

I’m with Hetzner at the moment, and have a dedicated server with them. Perhaps they will be accommodating, but if they cant or wont get the hardware, it seems like I’m just ■■■■ out of luck.

As far as the OP explained that already happened and there were still issues.

Were you in my position Sandro, what would you do?

This? This sounds like the server hardware. I still question why UDP is even let through.

No, in the other thread it was mentioned that the NIC flooding was stopped by the network firewall but there were still issues as the whole network was saturated.

That is a good question, the Hetzner firewall was disabled, but iptables was still enabled only allowing TCP packets from Cloudflare. I don’t have any deny rules though, I assumed anything that wasn’t whitelisted was dropped automatically.

1 Like

So that’s the host’s problem. Nothing a user can do other than wait for the host to null route any traffic to that IP address. Not a problem with Argo tunnel.

I believe the last rule should drop anything not covered by the rules.

Argo wouldn’t be the issue but it wouldn’t fix it either.

Really I just don’t understand what the attacker is doing to get my origin IP. I get being lucky, but this is the 3rd IP address that’s been discovered so far.

I’ll try and speak to the Hetzner NOC again tomorrow, (they’re all in bed now) and see if I can get them to do something on their infrastructure to just deny anything that isn’t Cloudflare

2 Likes

That’s why they’d need a bit of help from the host.

2 Likes

First question, what kind of change was it? x.x.x.5 → x.x.x.100 or something more profound? In other words, did you stay in the same network or did you change network altogether as well?

1 Like