Droplet CPU Usage 100% if Under Attack Mode is off!

When I turn off my Under Attack mode, the CPU usage goes constant at 100%. If I turn it back on it stays to normal.

Any idea why this is happening?

May be you are under attack, check your firewall logs instantly. What’s the domain

2 Likes

https://gofresh.com.kw

Is your Website hosted on a shared web hosting or a VPS, or rather a dedicated server?

Okay, you are running a WordPress Website.

I would suggest tuning a bit, if possible, your origin host / server PHP, etc. like below:

memory_limit = 256M
max_execution_time = 300
max_input_time = 1000
max_input_vars = 5000 or 7000
post_max_size = 16M
upload_max_filesize = 16M

Also, if you can install the PHP OPcache module as it could help too :wink:

Do not forget to save and restart your php service to apply the changes.

Furthermore, are the options like Minify (HTML, CSS, JavaScript), Brotli, HTTP/2, Rocket Loader and similar enabled in the Cloudflare dashboard?

Have you analyzed your Web traffic using?
Is it bots, crawlers, or DDoS?
At the Cloudflare dashboard, the Firewall Events or Analytics/Traffic tab could to determine and get something from it, if so.

Is Cloudflare allowed to connect to your origin host / server?

Bots…lots of bots. Probably. Your logs, or local analytics, should shed some light on what’s going on. Server logs will tell you all about what’s going on if you or your host can analyze them.

1 Like

I checked the server stats using “htop” and it turns out php-fpm is taking 100% CPU usage.

How to allow REST API calls when I am in under attack mode? I have WooCommerce website and I use REST API to connect it to my app.

Now I am in under attack mode but now my app show 403 error code when browsing.

How can I allow this?

IAUM is essentially a Firewall rule that JS Challenges everybody, but allows Verified/Known bots through.

You can create a Firewall rule like the above, but add a “AND NOT” statement for the API URL. (Updated rule screenshot)

2 Likes

Thank you for your reply

I checked the server stats using “htop” and it turns out php-fpm is taking 100% CPU usage.

That’s a lot of busy CPUs. Is your site normally very busy? It’s possible it’s just being attacked.

Have you looked at the server logs to see what’s getting hit so often? If it’s the API endpoint, you may have to tighten up that firewall rule.

When you had Under Attack mode enabled, was CPU usage lower?

1 Like

There are currently no visitors at the moment. There is no API calls.

And yes, when Under Attack is on then the CPU stays normal

How about this:

1 Like

Have you tried restarting php-fpm process?

A little bit out of the scope for this forums, but may I ask how did you setup and how do you use php-fpm? On demand, dynamic, static?

As @sdayman pointed out to check server log files, as it could be possible something is hitting xmlrpc.php or wp-login.php (very likely, known for WordPress) if so.

1 Like

If it’s bots hitting xmlrpc or wp-login, the JS Challenge should stop that.

We had an issue like this once. Our problem was that due to a bug in the code our server was sending out too many external HTTP requests that were taking a long time to complete. That way, we had far too many PHP-FPM processes running, thereby overloading the server.

Check which scripts your PHP-FPM is running. That might help.

1 Like

With Linux servers and reported CPU load averages, context matters i.e. how many CPU threads are available to serve the load.

What OS you using Ubuntu/CentOS ? Do you have sysstat package installed ?

yum -y install sysstat

apt-get -y install sysstat

Then you can see pidstat command to dig into server resource usage per process

To calculate php-fpm CPU usage per CPU thread

cpu_pc=$(($(nproc)*100))
pidstat -ulh -C php-fpm 1 1| egrep -v 'UID|Linux' | awk -v cpumax=$cpu_pc '{cpu += $7} END {print cpu"%",cpu/cpumax}'

example output from pidstat command

1126.86% 0.176072

This is on a 32 CPU core/64 CPU thread server running php-fpm where pidstat reports the cumulative CPU % usage per php-fpm process as being 1126.86% over 64 CPU, threads. Which reveals a per CPU thread php-fpm CPU usage of 0.176 x 100 = 17.6% which is in fact low CPU load usage as per CPU thread hasn’t hit 100% yet. For 64 CPU thread server, you’d be maxing out when CPU load reaches 64x100 = 6400%

And actual per process (php-fpm resource usage) via pidstat command for 1 second interval. The php-fpm processes individually only consume a small amount of cpu threads and for a 64 cpu thread server is very little at ~10-20% average cpu %.

pidstat -ulh -C php-fpm 1 1 | sed -e "s|$HOSTNAME|hostname|g"
Linux 3.10.0-1160.21.1.el7.x86_64 (hostname)    12/08/2021      _x86_64_        (64 CPU)

#      Time   UID       PID    %usr %system  %guest    %CPU   CPU  Command
 1638930016  1000     36202    1.90    0.00    0.00    1.90    27  php-fpm: pool www                                                                                               
 1638930016  1000     36207    1.90    0.00    0.00    1.90    31  php-fpm: pool www                                                                                               
 1638930016  1000     36208   53.33    1.90    0.00   55.24    25  php-fpm: pool www                                                                                               
 1638930016  1000     36211   34.29    2.86    0.00   37.14    12  php-fpm: pool www                                                                                               
 1638930016  1000     36212   43.81    3.81    0.00   47.62    22  php-fpm: pool www                                                                                               
 1638930016  1000     36215    5.71    1.90    0.00    7.62    56  php-fpm: pool www                                                                                               
 1638930016  1000     36216   96.19    0.95    0.00   97.14     9  php-fpm: pool www                                                                                               
 1638930016  1000     36218    0.95    0.00    0.00    0.95    19  php-fpm: pool www                                                                                               
 1638930016  1000     36230    0.95    0.00    0.00    0.95    48  php-fpm: pool www                                                                                               
 1638930016  1000     36233   46.67    2.86    0.00   49.52    56  php-fpm: pool www                                                                                               
 1638930016  1000     36237    5.71    0.00    0.00    5.71    28  php-fpm: pool www                                                                                               
 1638930016  1000     36239   43.81    4.76    0.00   48.57    15  php-fpm: pool www                                                                                               
 1638930016  1000     36241    2.86    1.90    0.00    4.76    63  php-fpm: pool www                                                                                               
 1638930016  1000     36248   20.00    0.00    0.00   20.00    13  php-fpm: pool www                                                                                               
 1638930016  1000     36251    6.67    1.90    0.00    8.57    46  php-fpm: pool www                                                                                               
 1638930016  1000     36254   97.14    0.95    0.00   98.10    34  php-fpm: pool www                                                                                               
 1638930016  1000     36263   47.62    3.81    0.00   51.43    25  php-fpm: pool www                                                                                               
 1638930016  1000     36271   33.33    2.86    0.00   36.19     6  php-fpm: pool www                                                                                               
 1638930016  1000     36273   61.90    3.81    0.00   65.71    51  php-fpm: pool www                                                                                               
 1638930016  1000     36278   27.62    0.00    0.00   27.62     1  php-fpm: pool www                                                                                               
 1638930016  1000     36290    0.95    0.00    0.00    0.95    56  php-fpm: pool www                                                                                               
 1638930016  1000     36292    2.86    0.95    0.00    3.81    31  php-fpm: pool www                                                                                               
 1638930016  1000     36294   17.14    0.00    0.00   17.14    20  php-fpm: pool www                                                                                               
 1638930016  1000     36296    6.67    0.95    0.00    7.62    23  php-fpm: pool www                                                                                               
 1638930016  1000     36299    5.71    1.90    0.00    7.62    30  php-fpm: pool www                                                                                               
 1638930016  1000     36302   81.90    2.86    0.00   84.76    26  php-fpm: pool www                                                                                               
 1638930016  1000     36304    4.76    0.00    0.00    4.76    26  php-fpm: pool www                                                                                               
 1638930016  1000     36305   28.57    2.86    0.00   31.43    21  php-fpm: pool www                                                                                               
 1638930016  1000     36306    4.76    0.00    0.00    4.76    27  php-fpm: pool www                                                                                               
 1638930016  1000     36308    3.81    1.90    0.00    5.71    61  php-fpm: pool www                                                                                               
 1638930016  1000     36311   13.33    0.00    0.00   13.33    13  php-fpm: pool www                                                                                               
 1638930016  1000     36316    0.95    0.00    0.00    0.95    26  php-fpm: pool www                                                                                               
 1638930016  1000     36318   22.86    2.86    0.00   25.71    26  php-fpm: pool www                                                                                               
 1638930016  1000     36319   11.43    1.90    0.00   13.33    10  php-fpm: pool www                                                                                               
 1638930016  1000     36321    0.95    0.95    0.00    1.90    18  php-fpm: pool www                                                                                               
 1638930016  1000     36323   73.33    0.00    0.00   73.33    36  php-fpm: pool www                                                                                               
 1638930016  1000     36325    1.90    0.00    0.00    1.90     6  php-fpm: pool www                                                                                               
 1638930016  1000     36326    1.90    0.00    0.00    1.90     3  php-fpm: pool www                                                                                               
 1638930016  1000     36328   21.90    0.95    0.00   22.86    62  php-fpm: pool www                                                                                               
 1638930016  1000     36330    9.52    0.00    0.00    9.52    40  php-fpm: pool www                                                                                               
 1638930016  1000     36333    1.90    0.00    0.00    1.90     7  php-fpm: pool www                                                                                               
 1638930016  1000     36335    1.90    0.00    0.00    1.90    16  php-fpm: pool www                                                                                               
 1638930016  1000     36337    0.95    0.95    0.00    1.90    52  php-fpm: pool www                                                                                               
 1638930016  1000     36346    3.81    0.95    0.00    4.76     8  php-fpm: pool www                                                                                               
 1638930016  1000     36349   57.14    0.00    0.00   57.14    11  php-fpm: pool www                                                                                               
 1638930016  1000     36351  100.00    0.00    0.00  100.00    28  php-fpm: pool www                                                                                               
 1638930016  1000     36352    1.90    0.00    0.00    1.90    46  php-fpm: pool www                                                                                               
 1638930016  1000     36356    6.67    0.00    0.00    6.67    63  php-fpm: pool www                                                                                               
 1638930016  1000     36357    0.95    0.00    0.00    0.95    27  php-fpm: pool www                                                                                               
 1638930016  1000     36361    1.90    0.00    0.00    1.90    22  php-fpm: pool www                                                                                               
 1638930016  1000     36363   76.19    4.76    0.00   80.95     0  php-fpm: pool www                                                                                               
 1638930016  1000     36368    5.71    0.95    0.00    6.67    10  php-fpm: pool www                                                                                               
 1638930016  1000     36382   28.57    0.00    0.00   28.57    60  php-fpm: pool www                                                                                               
 1638930016  1000     36383    0.95    0.95    0.00    1.90    14  php-fpm: pool www                                                                                               
 1638930016  1000     36384   23.81    0.00    0.00   23.81    16  php-fpm: pool www                                                                                               
 1638930016  1000     36387    6.67    1.90    0.00    8.57    51  php-fpm: pool www                                                                                               
 1638930016  1000     36396   27.62    3.81    0.00   31.43    14  php-fpm: pool www  

Basically, ensure your config php-fpm settings optimal and ensure the server specs you choose allow you to optimally serve your php-fpm load whether it be an attack or legit high traffic load. Cloudflare can help once you figure out if it’s legit or an attack as you can create Cloudflare firewall rules to eliminate attack traffic.

For optimal php-fpm settings you’d want to figure out your optimal php-fpm max children to CPU thread ratio and measure your php-fpm latency response times to figure the best ratio.

Example for my LEMP stack users for this server, I have a custom cminfo command for php-fpm mem/children ratio stats

cminfo phpmem

------------------------------------------------------------------
Total PHP-FPM Master Processes: 1
PHP-FPM Pool Names:
www
------------------------------------------------------------------
36201 php-fpm: master /usr/local/etc/php-fpm.conf
------------------------------------------------------------------
Current Free Memory (KB): 31729824
PHP-FPM Available Memory (KB): 40132380
Estimated Max PHP Children: 859.718
Estimated Max PHP Children To CPU Thread Ratio: 13.4331
PHP-FPM Total Children: 180 from 1 PHP-FPM master(s)
PHP-FPM Total Used Memory (KB): swap:0 uss:5124032 pss:5138232 rss:8402556
PHP-FPM Average Per Child (KB): swap:0 uss:28466.8 pss:28545.7 rss:46680.9
uss = user set size
pss = process set size
rss = resident set size

Also if your php-fpm binary is built with systemd stats reporting, look for php-fpm status line in systemctl status php-fpm output on the Status: output line

systemctl status php-fpm
● php-fpm.service - PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/php-fpm.service.d
           └─limit.conf
   Active: active (running) since Wed 2021-12-08 00:02:39 UTC; 2h 32min ago
 Main PID: 36201 (php-fpm)
   Status: "Processes active: 11, idle: 169, Requests: 364056, slow: 12, Traffic: 33.7req/sec"
   CGroup: /system.slice/php-fpm.service
Status: "Processes active: 11, idle: 169, Requests: 364056, slow: 12, Traffic: 33.7req/sec"
4 Likes

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.