Under an origin server DDoS - How are attackers finding my origin IP?

Just to double check, the machine was not compromised?

2 Likes

Not that I can see, I’m not sure why they wouldn’t go for the crown jewels AKA my database if they had some kind of shell. I do scan the system occasionally with chkrootkit and Lynis

I also use Hetzner dedis. You deny it using iptables at your dedi, or even using the Hetzner firewall interface for your dedi.

Possible reason is because you, I, they, got the server which was already used by some other people, which obviously exposed their IP - assume the “ex customer” did not use Cloudflare? (for web).
Could be a reason if the DNS history (PTR) for the given IP address(es) are somewhere saved and then the “bad guys” using hosts like DigitalOcean, Linode, etc. just “do their job as is”.

Moreover, you cannot hide from “legal scanners” like Censys, Stretchoid, Shodan, etc.

See the everyday available scans even here:

ICMP’s and pings are always running.

Have a chkrootkit and rkhunter just in case, hopefully your host is not hacked or having some malware running on.

Other possible options, pay a DDoS service from Voxility, SwissmadeHost, JavaPipe or just get another VPS in front of your dedi which would act like a firewall. Install PfSense on it and keep on going using the Cloudflare service for your needs :slight_smile:

And hopefully your DNS at Cloudflare does not expose some of the records (SPF, MX …), even your SSL certificate can do it (if we somehow bypass Cloudflare for some domain and connect directly to your host SSL).

The address will most certainly have been used already, but the understanding is it always is the same attacker.

Well, the machine is presumably locked down, so they can’t get anywhere either.

1 Like

I use both to deny. Either my NIC gets flooded, or Hetzners does it seems.

Maybe? But I had the 1st IP address for months with most of my DDoS attacks coming though Cloudflare, making them pretty easy to stop. My IP address isn’t listed in Shodan, or any other of the legal scanners.

This could work, but due to the sheer volume of data I push per month most services are very expensive.

I don’t think you can connect a VPS with a dedi using Hetzner, but I’m not sure. Wouldn’t that just move the issue anyway, so the VPS would go down?

I’m not sure I follow this point?

Can you install HetrixTools (or some other like Zabbix) as a simple monitoring and activate notifications (email, telegram) when you receive or send more than 100/300/500Mbps on network? (Hetzner has some traffic notifications too possible to setup if you exceed your daily/week/month traffic)

If you have a lot of IP’s to block, even ipset’s, just to note here, I hope you have at least 1G NIC at your dedi and a good CPU which can handle that.

A good XDP and iptables at 1G NiC can handle packets approx. like 10G NiC - I have tested that one for sure.

But how about 8443 and 8080 as alternative ports from 80 and 443 (even not open, but if running Nginx they are even if no app is running on them)?

What are your iptables config?
Is your dedi server a game server or?
Do you use any web sockets or WebRTC or Memcached?

I am afraid not. Anything passes until you setup the rules first.

Or someone just … well, allocated IP subnets are public for each ASN, so attacker can just go for each IP and boom. If ours are in that one, we got it too.

How about if contacting Hetzner support? To they see something more regarding your issue?

I have a good CPU and a 1 gig NIC, and I have traffic notifications setup within Hetzner Robot.

As for ports, I’ve conducted a few TCP port scans using a VPN to try and imitate an attacker, I do not get any results back, because any relevant open ports are only open to specific IP addresses and ranges.

I have internal ports that run on 8090 and a few others, but they are proxied through Nginx and don’t communicate externally. Yes I use WebSockets, but not WebRTC.

Hetzner support have been pretty good, but the NOC team take quite a long time to reply, but they are able to block the attack, at least temporarily.

1 Like

What about abuseipdb?:

Have you made any changes in your /etc/sysctl.conf regarding broadcast and your IPv4 and IPv6?
Just to note, an IPv6 address you got “by default” at Hetzner - is it available to public and configured at network interfaces and/or sysctl.conf (hopefully disabled)?

Just wonder, do you use vSwitch at Hetzner?

My current IP is not in the AbuseDB

No, I didn’t know those were broadcast at all

I am currently not using IPv6 at all. I don’t have one with Hetzner.

No, I’ve never needed to use VLANs

1 Like

A post was merged into an existing topic: DDOS Attack from thousands of USA IP’s

Mailgun will leak your server IP as it passes your real server IP in mail headers. As @cscharff stated you need to strip the real server IP from mail headers yourself or use a 3rd party SMTP transactional outbound mail provider that does this automatically. Amazon SES is one that does strip real server IPs from relayed/sent SMTP outbound email. If you’re behind Cloudflare and send outbound mail, Amazon SES is a must as one of many steps to reduce real server IP leakage.

1 Like

@PeterPennywacker Argo Tunnel should be free and separate from Argo if you have a Cloudflare For Teams free subscription. I did a write up at https://blog.centminmod.com/2021/02/09/2250/how-to-setup-cloudflare-argo-tunnel-on-centos-7/

3 Likes

Thanks for all the info everyone, here’s what I’ll try:

  • Further lock down outbound connections from origin
  • Check /etc/sysctl.conf regarding the broadcast of v4 and v6 addresses.
  • I don’t use Mailgun anymore, but I’ll close my account with them just in case.
  • Perform some more AV and rootkit scans.
  • Add block rules for inbound connections, including ICMP

I do have a TLSA record on my Cloudflare account, there’s no way that would be leaking my IP right?

A TLSA record contains a hash and no address, so that should not be the reason.

As mentioned earlier, controlling outbound connections might confirm that this was the issue, but it might also break things.

If you are not using Mailgun that should not leak the address either and broadcasts should neither be a problem as this only involves the local network.

I know, I mentioned it before but I’d even rule out ICMP, though you can still block it. IMHO the most likely reason will be outbound connections, but of course that really depends on your setup and what your machine is configured to do. A close second might be if your machine was compromised but that’s also speculation at this point.

You really can only go through all of that one by one and make sure the machine does not communicate its address in any way. Though at this point we really are far beyond the scope of the forum :slight_smile:

2 Likes

As a quick update, just had this from Hetzner regarding the issues:

we have found the issue and resolved it.
It was an attacker that attacked your Hetzner IP and didn’t go over your Cloudflare IP. As to why our mitigation didn’t catch it, we are investigating. But rest assured that we have taken swift and decisive action against the attacker.

1 Like

Ehm, what’s the update here? :smile:

1 Like

I guess admitting their mitigation wasn’t working? Perhaps that’s why when I have their firewall enabled, requests on my origin return to normal, but the frontend website still displayed numerous connection issues.

That, sure and it might be nice that their own security layers might be working now but it won’t address the actual issue of your address leaking.

The only thing now might be that your link might not be saturated any more.

2 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.