Server load spikes from CF?

I’ve been dealing with server spikes ever since using Ezoic (a Cloudflare partner). I moved everything to Cloudflare so that I would have access to WAF and other rules, but I’m still dealing with spikes.

Every time I see a spike, it looks like the culprit is a Cloudflare IP. This is from WHM’s Apache Status, sorted by CPU, taken shortly after WHM or Apache shut down the offending PID 9816:

PID Acc M CPU SS Req Dur Conn Child Slot Client Protocol Request
9816 1/305/16930 G 152.36 0 44 403203 21.4 6.06 332.65 172.70.174.61 h2 [1/1] done
9816 1/306/16736 G 152.36 0 1 361284 21.7 5.64 332.16 172.70.42.237 h2 [1/1] done
9816 0/310/16878 G 151.37 1 193 468429 0 10.97 338.37 172.70.43.81 h2 [1/0] read: stream 0,

These 3 were by far the highest CPU; the 4th line had a load of 44.27 from a local IP.

Unsorted, I see a load of 3.63 with one connection and then a load of 21.63 with the next connection, and the load keeps going up from there. And both IPs belong to Amazon:

12486 0/36/16425 _ 21.63 0 28 264878 0 0.72 318.63 52.70.240.171 http/1.1
12794 0/6/15879 _ 3.63 7 27 317376 0 0.15 315.86 52.70.240.171 http/1.1

Any thoughts or suggestions?

This is definitely the problem, and I don’t know how to fix it.

I’ve been fielding server spikes ALL day! Literally every 10-15 minutes. Most recently it hit 159! Obviously my sites were all crashing.

I have sys-snap.pl running, so I was able to look at the process list after I restarted Apache. At the time of the spike I had 508 active connections.

I copied them to Excel to de-dupe the IPs, and that left 378.

109 of the remaining IPs belong to Amazon.

256 belong to Cloudflare.

I have a rule set up that requires an Interactive Challenge, so I added this to it:

lower(http.host) contains "cloudflare" or
lower(http.host) contains "amazon" or
[the rest of the original expression]

Since adding that, I’ve gone almost an hour with no more spikes.

This certainly doesn’t feel like a smart solution, but Rate Limit doesn’t let me use http.host. I already use cf.bot_management.verified_bot in a Rate Limit rule, but apparently CF doesn’t include itself in the managed list.

What do I do? Add Amazon and Cloudflare to the Verified Bot list?

Well, I just spiked again, so I don’t think that rule had the impact I’d hoped. This time I had 1,163 connections; after de-duping, I had 388 unique IPs.

Of those, 148 belong to Amazon and 235 belong to Cloudflare.

I tried blocking entirely if http.host contains cloudflare or amazon, but that seems like a VERY bad idea! I don’t know how else to get it under control, though.

I have CSF (ConfigServer Firewall), so I enabled CT_LIMIT there and set it to limit an IP on port 80,443 to 100 connections for 5 minutes.

Any other suggestions?

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.