Warning: Attacks are bypassing cloudflare!

I just wanted to inform cloudflare security team and everyone else. Currently some people are bypassing the entire cloudflare layer 7 protection. Below are the logs from the nginx and attack. The server is trusted to cloudflare ips only(IP Ranges | Cloudflare) with iptables + datacenter firewall and the domain is proxied by cloudflare for sure. So there is no way to access to server with an IP address without domain hosted in cloudflare. This has been tested already before I report here. Unless they are spoofing cloudflare ips which I dont think ?

103.22.201.230 - - [29/May/2021:23:45:56 +0200] “GET /clients.php HTTP/1.1” 200 “https://www.domain.com/clients.php” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36” “230.214.242.233, 99.108.240.110,122.2.28.114”

Above nginx log clearly shows the request made over cloudflare with cloudflare IP 103.22.201.230 and the attacker layer 7 bot ips are in x_forwarded_for are below
230.214.242.233
99.108.240.110
122.2.28.114

Above is just one log line, there are over 100.000 records with million different ips, even the 230.214.242.233 is not assigned to any network or country. All requests are coming over cloudflare with various cloudflare ips. I dont know how but they simply found a way to bypass cloudflare, it looks like a very recent bypass and coming hard to every other website if the cloudflare does not take any action before it happens.

The log clearly shows the connections are made over cloudflare proxy and those bot ips are passing the cloudflare servers now. I recommend the cloudflare network team to watch the entire network and connections.

I am not clear on what you posted - are you saying the visits are all showing the whitelisted Cloudflare IP addresses but x_forwarded_for are from different IP? If that’s the case, then it’s correct. If you want your logs to show the visitor address instead, read Restoring original visitor IPs – Cloudflare Help Center

If that’s not the case - how do you know these are bots and not actual visitors?

Update on this issue:

We have noticed that the attackers are found our server IP somehow, however the main problem is they are spoofing the cloudflare’s trusted ip range with their own attack servers. I dont know how but we have confirmed that they were calling the http://serverip of the server to make the attack, since the server was already configured to accept only cloudflare trusted ips and they are spoofing the entire cloudflare ip range https://www.cloudflare.com/ips/

So anyone out there configured their server to just allow cloudflare trusted ips on their server or on network with iptables firewall etc, just to be informed that attackers are spoofing cloudflare ips on layer7 attack type and the connections are exactly appear as coming from cloudflare trusted ips which deceives the iptables or the firewall and server accepts the connections as it thinks the connection is coming from trusted cloudflare ips. This is a new type of bypassing cloudflare and effective when they find your server ip, we did kept the server ip secret like nasa security, no dns records, nothing shows ips, not in emails etc… but they somehow got it.

To prevent the attack you need to implement something on your webserver such as nginx to block direct ip access http://serverip and allow only https://domain.com access. This is what we did to solve the issue, it is not a complete solution since the datacenter may suspend or null route your server ip due to high volume of attack coming even if your server is not affected. But it can lead the attacker to get bored. How they find the website server ip behind cloudflare is another subject, I truly no idea on that.

This is very difficult to achieve. Do you have any data to show that it is spoofed?

There are various things you can do to stop this. Ask your hosting provider if they support RPKI with drop invalid. That would stop this immediately. They should also support BCP 38.

On your Cloudflare domains, enable Authenticated Origin Pull with custom certificates. This prevents anything except your Cloudflare account from talking to your Origin.

2 Likes

Thank you for the update. Its a dedicated server and the only data I can show is the nginx access.log and it already shows the spoofing. I did also check it with tcpdump and connections are coming from cloudflare ip range however directly to http://serverip instead of https://domain.com and this is impossible, since the server firewall is configured to allow only cloudflare ips, but when they spoof the cloudflare ips the server accepts requests to http://serverip

103.22.201.230 - - [29/May/2021:23:45:56 +0200] “GET /clients.php HTTP/1.1” 200 “https://www.domain.com/clients.php ” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36” “230.214.242.233, 99.108.240.110,122.2.28.114

Above is just one log line from nginx, it absolutely looks like a legit connection coming from cloudflare proxy server 103.22.201.230 and true client/browser ips are below which are sent by cloudflare

230.214.242.233, 99.108.240.110,122.2.28.114

These above ips are the source of bot servers and they make requests from spoofing source which is 103.22.201.230.

Below is the screen image from the nginx access.log, I stop it to take a screen image. You can see on the left side all connections are coming from cloudflare ips, those are all cloudflare ips and on the right side you can see the client/browser true ips which are sent by cloudflare in CF-Connecting-IP header. There are thousands of them when there is an attack. Unless they are bypassing directly cloudflare proxy, this is ip spoofing and direct access to http://serverip with spoofed cloudflare ips as the server is trusted to cloudflare ips only. I can send you the domain on cloudflare if you guys want to investigate it by private email or message if possible. I deleted the sensitive part in the picture, the nginx is returning 444 to those all requests now by rate limiting and if called http://serverip

It’s plain impossible, you can’t establish a connection with an ip spoofed header and thus would make the webserver never be able to finish an HTTP request. At most it can accept an incoming connection that timeouts, but that’s about it.

1 Like

Then someone has a very powerful layer7 attack type that bypasses cloudflare proxy if not possible to spoof as you describe. It even bypasses active im under attack mode + bot fight mode. Check my above new reply to see the screen image, all connections are from cloudflare ips clearly and proxied client ips are all bot and botnet servers, I checked few of them.

Note that Cloudflare is not self-managed, you need to tune your rules to prevent Layer7 attacks effectively. The under attack mode is only a last resource that won’t stop a dedicated/experienced attacker, bot fight mode is made to prevent unwanted bots, chances are that if they can solve the js challenge they can also go through the bot fight mode.

You need to look into captchas, if that doesn’t work your only chances are either rate limit & block or to invest in a more complex Layer7 mitigation mechanism.

1 Like

It is not impossible, but you need to spoof the IP advertisement into the network to establish connections. For most attackers it is almost impossible. With RPKI it is still theoretically possible, but in the real world I doubt that we would see such an attack on these forums.

@shaqun5 have you enabled Authenticated Origin Pull?

1 Like

@michael
I will check the “Authenticated Origin Pull” today, I couldnt find a time for it.

@jnperamo I already took measures after this issue and the server is rate limiting perfectly now but it isnt a solution at all. Usually almost all datacenters will null route the server ip or suspend the entire server because the attack passes their router and network before reaches to the server even the server itself does not affected at all with total load %1.2. I will check other solutions as best as I can.

You can use Cloudflare rate limit to prevent that load from your servers. However, null routing only makes sense on a network attack, not on an application attack, I have never seen an application attack overwhelm a network, if that’s your case (which I doubt) you will most likely be asked to upgrade to the enterprise plan.

Its the server provider who null routes or suspends the server completely when the attack bypasses the cloudflare and reaches to their network, not the cloudflare. Most of the server providers available out there null routing the server ip for 3 days or suspends the server for 5 days on layer7 attacks, they dont care if you do a rate limiting on your server and if your server is perfectly fine, they care their own routers and network, because none of them has a true layer7 protection at all, even cloudflare is bypassed as noticed today with my case and I just learned from you that it can be bypassed by a dedicated attacker. I will try the “Authenticated Origin Pull” method today as Michael suggested when i have time.

That is why I need to fix the issue before it reaches to server.