522 Error during high traffic load - Database tuning?

When we have a high number of visitors trying to purchase products on Woocommerce (a Wordpress plugin) a 522 error occurs. Not only for several of my customers but also for me when I’m accessing the admin area of Wordpress.

Server setup: VPS - CentOS 7 - Plesk Onyx, CPU:6 vCore, RAM:12 GB, SSD:240 GB

Is it possible this is simply a mySQL max_connection problem? It defaults to 151. I changed it to 300 within phpMyAdmin, and then I did a restart (# service mariadb restart) to solidify all the setting changes. Unknowingly, if I don’t make that setting change in the my.cnf file that it will default back to 151. So, each time I have experienced this problem, max_connections has been set to 151 instead of 300. I imagine there could be more than 151 connections if there are about 200+ people trying to purchase a product at the same time. So this could really be the problem. The load on the server seems low as shown in the following images. The spike is when the 522 error occurred. (new users can only show one photo so I had to remove these images)

  • 5% cpu usage spike during event
  • 700M memory used of 12GB during event

I have gone through this resource but I have not found anything here that would cause this problem: https://www.ionos.com/digitalguide/hosting/technical-matters/error-522-explanation-and-solutions/

  • Plesk Firewall was not enabled during the event.
  • There was no rate limiting by my host.
  • Keep-Alive is enabled.
  • If I was filtering Cloudflare IP I would think that the website would not operate.

Even if Max Connections was the problem, I would still like to optimize my mysql variables to prevent further slow connections for my customers during peaks like this. I would appreciate if someone could check the settings in the file below plus a recent run I did with MySQLTuner.

Here is my /etc/my.cnf file:

[mysqld]
bind-address = ::ffff:127.0.0.1
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
innodb_buffer_pool_size=1028M
query_cache_size=64M
**# The next line for Max Connections is new and untested.**  
**max_connections=400**
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in 

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

Here is the results of MySQLTuner:

[[email protected] ~]# ./mysqltuner.pl
 >>  MySQLTuner 1.7.19 - Major Hayden <[email protected]>
 >>  Bug reports, feature requests, and downloads at 
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[!!] Your MySQL version 5.5.64-MariaDB is EOL software!  Upgrade soon!
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ----------------------------------------------                                              --------------------
[OK] Log file /var/log/mariadb/mariadb.log exists
[--] Log file: /var/log/mariadb/mariadb.log(10K)
[OK] Log file /var/log/mariadb/mariadb.log is readable.
[OK] Log file /var/log/mariadb/mariadb.log is not empty
[OK] Log file /var/log/mariadb/mariadb.log is smaller than 32 Mb
[OK] /var/log/mariadb/mariadb.log doesn't contain any warning.
[OK] /var/log/mariadb/mariadb.log doesn't contain any error.
[--] 7 start(s) detected in /var/log/mariadb/mariadb.log
[--] 1) 191026  1:39:02 [Note] /usr/libexec/mysqld: ready for connections.
[--] 2) 191011 19:15:23 [Note] /usr/libexec/mysqld: ready for connections.
[--] 3) 191011 19:13:27 [Note] /usr/libexec/mysqld: ready for connections.
[--] 4) 190919  3:42:01 [Note] /usr/libexec/mysqld: ready for connections.
[--] 5) 190703 19:14:24 [Note] /usr/libexec/mysqld: ready for connections.
[--] 6) 190703 18:59:52 [Note] /usr/libexec/mysqld: ready for connections.
[--] 7) 190703 18:59:46 [Note] /usr/libexec/mysqld: ready for connections.
[--] 6 shutdown(s) detected in /var/log/mariadb/mariadb.log
[--] 1) 191026  1:39:00 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 2) 191011 19:15:22 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 3) 191011 19:13:26 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 4) 190919  3:42:00 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 5) 190703 19:13:07 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 6) 190703 18:59:50 [Note] /usr/libexec/mysqld: Shutdown complete

-------- Storage Engine Statistics ---------------------------------------------                                              --------------------
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MEMORY +MRG_MYIS                                              AM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 186.8M (Tables: 110)
[--] Data in InnoDB tables: 14.5M (Tables: 374)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics ------------------------------------------                                              --------------------
[--] innodb_stats_on_metadata: ON
[!!] Stat are updated during querying INFORMATION_SCHEMA.

-------- Security Recommendations ----------------------------------------------                                              --------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] User '[email protected]%' does not specify hostname restrictions.
[!!] There is no basic password file list!

-------- CVE Security Recommendations ------------------------------------------                                              --------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics ---------------------------------------------------                                              --------------------
[--] Up for: 1d 21h 27m 31s (5M q [31.264 qps], 73K conn, TX: 16G, RX: 692M)
[--] Reads / Writes: 96% / 4%
[--] Binary logging is disabled
[--] Physical Memory     : 11.6G
[--] Max MySQL memory    : 1.8G
[--] Other process memory: 0B
[--] Total buffers: 1.3G global + 2.8M per thread (151 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 1.4G (12.08% of installed RAM)
[OK] Maximum possible memory usage: 1.8G (15.21% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory                                               available
[OK] Slow queries: 0% (0/5M)
[OK] Highest usage of available connections: 11% (18/151)
[OK] Aborted connections: 0.00%  (1/73200)
[!!] name resolution is active : a reverse name resolution is made for each new                                               connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[OK] Query cache efficiency: 35.8% (2M cached / 7M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 217K sorts)
[!!] Joins performed without indexes: 7704
[!!] Temporary tables created on disk: 50% (92K on disk / 182K total)
[!!] Thread cache is disabled
[!!] Table cache hit rate: 18% (400 open / 2K opened)
[!!] table_definition_cache(400) is lower than number of tables(587)
[OK] Open file limit used: 24% (233/962)
[OK] Table locks acquired immediately: 99% (2M immediate / 2M locks)

-------- Performance schema ----------------------------------------------------                                              --------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ----------------------------------------------------                                              --------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 6 thread(s).
[--] Using default value is good enough for your version (5.5.64-MariaDB)

-------- MyISAM Metrics --------------------------------------------------------                                              --------------------
[!!] Key buffer used: 61.7% (82M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/55.8M
[OK] Read Key buffer hit rate: 99.8% (37M cached / 65K reads)
[!!] Write Key buffer hit rate: 91.2% (384K cached / 350K writes)

-------- InnoDB Metrics --------------------------------------------------------                                              --------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[!!] InnoDB File per table is not activated
[OK] InnoDB buffer pool / data size: 1.0G/14.5M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (0.972762645914397 %):                                               5.0M * 2/1.0G should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 99.84% (880634 hits/ 882005 total)
[!!] InnoDB Write Log efficiency: 59.2% (1795 hits/ 3032 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1237 writes)

-------- AriaDB Metrics --------------------------------------------------------                                              --------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/1B
[!!] Aria pagecache hit rate: 88.5% (305K cached / 35K reads)

-------- TokuDB Metrics --------------------------------------------------------                                              --------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics --------------------------------------------------------                                              --------------------
[--] XtraDB is disabled.

-------- Galera Metrics --------------------------------------------------------                                              --------------------
[--] Galera is disabled.

-------- Replication Metrics ---------------------------------------------------                                              --------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: STATEMENT
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server

-------- Recommendations -------------------------------------------------------                                              --------------------
General recommendations:
    Restrict Host for 'dboxxxxxxx'@% to [email protected]
    UPDATE mysql.user SET host ='SpecificDNSorIp' WHERE user='dboxxxxxxx' AND                                               host ='%'; FLUSH PRIVILEGES;
    Enable the slow query log to troubleshoot bad queries
    Configure your accounts with ip or subnets only, then update your configurat                                              ion with skip-name-resolve=1
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes                                               are found.
             See 
             (specially the conclusions at the bottom of the page).
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Set thread_cache_size to 4 as a starting value
    Increase table_open_cache gradually to avoid file descriptor limits
    Read this before increasing table_open_cache over 64: 
    Read this before increasing for MariaDB                                             timizing-table_open_cache/
    This is MyISAM only table_cache scalability problem, InnoDB not affected.
    See more details here: 
    This bug already fixed in MySQL 5.7.9 and newer MySQL versions.
    Beware that open_files_limit (962) variable
    should be greater than table_open_cache (400)
    For MySQL 5.6.2 and lower, Max combined innodb_log_file_size should have a c                                              eiling of (4096MB / log files in group) - 1MB.
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read t                                              his: 
Variables to adjust:
    SET innodb_stats_on_metadata = OFF
    query_cache_size (=0)
    query_cache_type (=0)
    join_buffer_size (> 128.0K, or always use indexes with JOINs)
    tmp_table_size (> 16M)
    max_heap_table_size (> 16M)
    thread_cache_size (start at 4)
    table_open_cache (> 400)
    table_definition_cache(400) > 587 or -1 (autosizing if supported)
    innodb_file_per_table=ON
    innodb_log_file_size should be (=128M) if possible, so InnoDB total log file                                              s size equals to 25% of buffer pool size.
[[email protected] ~]#

Thank you in advance for any assistance you can offer.

Regards,
Mark

I changed Max Connections to 400 and did not get an error during our latest visitor spike of 270. Each trying to add a product to the cart and checking out. However, it did run slow with warnings regarding the Apache & php-fpm memory usage. So my guess is php-fpm tuning is required. Below are the memory and cpu usage. PHP-FPM used 4.6GB out of 10GB available. The CPU got to about 75% during this spike.

image%20(6) image%20(7)

This article and this one discuss tuning php-fpm. My settings are:

  • listen.allowed_clients = 127.0.0.1
  • pm = dynamic
  • pm.max_children = 50
  • pm.start_servers = 5
  • pm.min_spare_servers = 5
  • pm.max_spare_servers = 35
  • pm.max_requests = 0

I ran a AB load test # ab -n 100 -c 10 https://mysite.com/cart
And I got back the following results. I tried changing the parameters but nothing really seemed to help much.

This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking ultracornhole.com (be patient).....done


Server Software:        nginx
Server Hostname:        ultracornhole.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256

Document Path:          /cart
Document Length:        112275 bytes

Concurrency Level:      10
Time taken for tests:   7.471 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      11269600 bytes
HTML transferred:       11227500 bytes
Requests per second:    13.38 [#/sec] (mean)
Time per request:       747.140 [ms] (mean)
Time per request:       74.714 [ms] (mean, across all concurrent requests)
Transfer rate:          1473.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    6   3.3      5      15
Processing:   403  686  74.8    689    1105
Waiting:      403  685  75.3    689    1105
Total:        418  691  74.6    694    1110

Percentage of the requests served within a certain time (ms)
  50%    694
  66%    698
  75%    702
  80%    704
  90%    720
  95%    774
  98%    875
  99%   1110
 100%   1110 (longest request)

I wonder if everything is tuned as optimal as possible, would the server still be slow with this many visitors.