Page 1 of 1
Mariadb doesn't restart
Posted: Mon Nov 18, 2019 7:17 am
by Teo
Hi guys, I have this problem, any help?
seems the system eat RAM untill the database crashes. And now won't restart anymore.
Thank you.
Code: Select all
191118 01:23:53 mysqld_safe mysqld restarted
191118 1:23:53 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
191118 1:23:53 [Note] /usr/libexec/mysqld (mysqld 5.5.64-MariaDB) starting as process 21649 ...
191118 1:23:53 InnoDB: The InnoDB memory heap is disabled
191118 1:23:53 InnoDB: Mutexes and rw_locks use GCC atomic builtins
191118 1:23:53 InnoDB: Compressed tables use zlib 1.2.7
191118 1:23:53 InnoDB: Using Linux native AIO
191118 1:23:54 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
191118 1:23:54 InnoDB: Completed initialization of buffer pool
191118 1:23:54 InnoDB: Fatal error: cannot allocate memory for the buffer pool
191118 1:23:54 [ERROR] Plugin 'InnoDB' init function returned error.
191118 1:23:54 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
191118 1:23:54 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
191118 1:23:54 [Note] Plugin 'FEEDBACK' is disabled.
191118 1:23:54 [ERROR] Unknown/unsupported storage engine: InnoDB
191118 1:23:54 [ERROR] Aborting
191118 1:23:54 [Note] /usr/libexec/mysqld: Shutdown complete
191118 01:23:54 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
191118 08:00:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
191118 8:00:56 InnoDB: The InnoDB memory heap is disabled
191118 8:00:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
191118 8:00:56 InnoDB: Compressed tables use zlib 1.2.7
191118 8:00:56 InnoDB: Using Linux native AIO
191118 8:00:56 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
191118 8:00:56 [Note] /usr/libexec/mysqld (mysqld 5.5.64-MariaDB) starting as process 25484 ...
191118 8:00:56 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
191118 8:00:56 InnoDB: Completed initialization of buffer pool
191118 8:00:56 InnoDB: Fatal error: cannot allocate memory for the buffer pool
191118 8:00:56 [ERROR] Plugin 'InnoDB' init function returned error.
191118 8:00:56 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
191118 8:00:56 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
191118 8:00:56 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
191118 8:00:56 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
191118 8:00:57 [Note] Plugin 'FEEDBACK' is disabled.
191118 8:00:57 [ERROR] Unknown/unsupported storage engine: InnoDB
191118 8:00:57 [ERROR] Aborting
Re: Mariadb doesn't restart
Posted: Mon Nov 18, 2019 9:45 am
by Teo
So I restarted the VPS and it restarted properly but I still have problems with RAM usage...
1st of all my system specs:
CentOS 7.7.1908(x86_64)
Kernel Version - Machine Architecture 3.10.0-1062.4.3.el7.x86_64 - x86_64
CPU Info Intel Core Processor (Haswell, no TSX), 2 cores
RAM 8 GB (no swap)
Installed VestaCP and I installed the php switcher with all versions.
HTTPD STATUS:
Server MPM: prefork
Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.16
3 websites are using php5.6 with REMI-php switcher
I did upgrade the OS because I used CentOS 6 before and I didnt have such problems with the default httpd vesta template. I only had if i switched to other httpd template like php7 and now with php5.6.
What I did untill now:
added:
Code: Select all
innodb_buffer_pool_size=64M
performance_schema = off
In
added:
Code: Select all
MaxKeepAliveRequests 500
KeepAliveTimeout 5
KeepAlive Off
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 3000
</IfModule>
HostnameLookups Off
Do I need to add something in /etc/my.cnf.d/mysql-clients.cnf (or server.cfg or client.cfg)? Or mybe something I forgot?
Any help it's much appreciated.
Thank you.
Re: Mariadb doesn't restart
Posted: Mon Nov 18, 2019 11:15 am
by mehargags
This can be because one or more of your sites are compromised or hitting high on MySQL queries which eventually crashes it "Out of Memory".
Please give more details, what kind of sites you host (wordpress or other CMS), how many sites, etc.
Also add SWAP, it helps enormously for MySQL
Re: Mariadb doesn't restart
Posted: Mon Nov 18, 2019 11:56 am
by Teo
mehargags wrote: ↑Mon Nov 18, 2019 11:15 am
This can be because one or more of your sites are compromised or hitting high on MySQL queries which eventually crashes it "Out of Memory".
Please give more details, what kind of sites you host (wordpress or other CMS), how many sites, etc.
Also add SWAP, it helps enormously for MySQL
Thank you for the reply. Always had 3 wordpress websites 2 phpnuke. Problem is I did not have any issue with the old configuration centos 6 + default httpd template set as before. Problem comes out if i chage httpd template and this is a must cause the new wordpress not supporting php5.4 anymore. i think is something missing in some of the config... Any idea?
This is the active memory usage:
Code: Select all
8001 m total memory
7720 m used memory
7508 m active memory
72 m inactive memory
139 m free memory
3 m buffer memory
138 m swap cache
0 m total swap
0 m used swap
0 m free swap
359807 non-nice user cpu ticks
0 nice user cpu ticks
92286 system cpu ticks
2760445 idle cpu ticks
52622 IO-wait cpu ticks
0 IRQ cpu ticks
6206 softirq cpu ticks
1761 stolen cpu ticks
21200833 pages paged in
9564836 pages paged out
0 pages swapped in
0 pages swapped out
12753786 interrupts
10397440 CPU context switches
1574061664 boot time
145025 forks
I run this script:
Code: Select all
curl -sL https://raw.githubusercontent.com/richardforth/apache2buddy/master/apache2buddy.pl | perl
had MaxRequestWorkers set to 255 now i add the line in the httpd.conf this way:
Code: Select all
MaxKeepAliveRequests 500
KeepAliveTimeout 5
KeepAlive Off
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 3000
MaxRequestWorkers 173
</IfModule>
HostnameLookups Off
added swap with this tutorial:
https://www.digitalocean.com/community/ ... n-centos-7
and now it says:
Code: Select all
total used free shared buff/cache available
Mem: 7631 4215 2195 26 1219 3108
Swap: 4095 0 4095
Total: 11727 4215 6291
--------------------------------------------------------------------------
8001 m total memory
4420 m used memory
4926 m active memory
389 m inactive memory
2302 m free memory
70 m buffer memory
1208 m swap cache
4294 m total swap
0 m used swap
4294 m free swap
504232 non-nice user cpu ticks
0 nice user cpu ticks
134770 system cpu ticks
3831739 idle cpu ticks
72853 IO-wait cpu ticks
0 IRQ cpu ticks
8813 softirq cpu ticks
2452 stolen cpu ticks
31682905 pages paged in
13055764 pages paged out
0 pages swapped in
0 pages swapped out
17918677 interrupts
14399569 CPU context switches
1574061664 boot time
201125 forks
--------------------------------------------------------------------------
Re: Mariadb doesn't restart
Posted: Tue Nov 19, 2019 8:16 am
by Teo
Update
after 24 hours I have this, for me is weird....
server usage
httpd
web server
CPU: 1.3Memory: 876 mb
Uptime: 19 hours
nginx
reverse proxy
CPU: 0Memory: 17 mb
Uptime: 1 day
named
dns server
CPU: 0Memory: 337 mb
Uptime: 1 day
mariadb
database server
CPU: 0.7Memory: 1609 mb
Uptime: 20 hours
fail2ban
brute-force monitor
CPU: 0.1Memory: 738 mb
Uptime: 1 day
for me something is wrong is impossible that I'm using 12 GB RAM as the graph shows.. any thought please?