Падает MySql InnoDB: Starting crash recovery

[ayaroshenko]

Добрый день. Очень нуждаюсь в помощи. Каждый день падает MySql.
Причину самостоятельно не сумел выявить.

Вот лог mysqld.log

SpoilerShow

Version: '5.5.55' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
180311 3:03:08 [Note] Plugin 'FEDERATED' is disabled.
180311 3:03:08 InnoDB: The InnoDB memory heap is disabled
180311 3:03:08 InnoDB: Mutexes and rw_locks use GCC atomic builtins
180311 3:03:08 InnoDB: Compressed tables use zlib 1.2.3
180311 3:03:08 InnoDB: Using Linux native AIO
180311 3:03:08 InnoDB: Initializing buffer pool, size = 128.0M
180311 3:03:08 InnoDB: Completed initialization of buffer pool
180311 3:03:08 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 46038706334
180311 3:03:08 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 46038732437
180311 3:03:08 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
180311 3:03:09 InnoDB: Waiting for the background threads to start
180311 3:03:10 InnoDB: 5.5.55 started; log sequence number 46038732437
180311 3:03:10 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
180311 3:03:10 [Note] - '0.0.0.0' resolves to '0.0.0.0';
180311 3:03:10 [Note] Server socket created on IP: '0.0.0.0'.
180311 3:03:10 [Note] Event Scheduler: Loaded 0 events
180311 3:03:10 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.55' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi

Вот конфиг mysql

SpoilerShow

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
skip-host-cache
skip-name-resolve

skip-external-locking
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 4
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 1M
net_buffer_length = 2K
thread_stack = 512K

#innodb_use_native_aio = 0
innodb_file_per_table

max_connections=35
max_connect_errors=50
max_user_connections=25
wait_timeout=10
interactive_timeout=50
long_query_time=5

#slow_query_log=1
#slow_query_log_file=/var/log/mysql-slow-queries.log

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

Вот лог messages в котором видно, что не хватает памяти. Вопрос как узнать куда она девается в этот момент ошибки? Что ее сжирает и процесс убивается?

SpoilerShow

Mar 11 03:02:26 python-studio fail2ban.filter[1295]: INFO [exim-iptables] Found 91.200.12.13
Mar 11 03:03:08 python-studio kernel: [5462598.330997] Out of memory in UB 64478: OOM killed process 27165 (mysqld) score 0 vm:2223588kB, rss:103240kB, swap:122808kB

[skurudo]

Тюнинговать конфиг для начала
viewtopic.php?f=32&t=14149

[ayaroshenko]

Тюнинговать конфиг для начала
viewtopic.php?f=32&t=14149

Спасибо за отклик! А что дает этот тюнинг? Что конкретно он делает?

И там их два, какой скачивать? Этот MySQLTuner или MySQL performance tuning prime ?

[skurudo]

Любой из них - они показывают, какие параметры завышены или наоборот.
Можно попробовать оба для сравнения.

[ayaroshenko]

Любой из них - они показывают, какие параметры завышены или наоборот.
Можно попробовать оба для сравнения.

Вот вывод такой дал первый скрипт

SpoilerShow

[OK] Currently running supported MySQL version 5.5.55
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/log/mysqld.log(1M)
[OK] Log file /var/log/mysqld.log exists
[OK] Log file /var/log/mysqld.log is readable.
[OK] Log file /var/log/mysqld.log is not empty
[OK] Log file /var/log/mysqld.log is smaller than 32 Mb
[!!] /var/log/mysqld.log contains 13583 warning(s).
[!!] /var/log/mysqld.log contains 2 error(s).
[--] 245 start(s) detected in /var/log/mysqld.log
[--] 1) 180401 0:02:20 [Note] /usr/libexec/mysqld: ready for connections.
[--] 2) 180331 6:11:09 [Note] /usr/libexec/mysqld: ready for connections.
[--] 3) 180331 4:55:10 [Note] /usr/libexec/mysqld: ready for connections.
[--] 4) 180329 3:04:46 [Note] /usr/libexec/mysqld: ready for connections.
[--] 5) 180329 2:03:46 [Note] /usr/libexec/mysqld: ready for connections.
[--] 6) 180328 2:23:09 [Note] /usr/libexec/mysqld: ready for connections.
[--] 7) 180328 2:09:10 [Note] /usr/libexec/mysqld: ready for connections.
[--] 8) 180327 5:11:15 [Note] /usr/libexec/mysqld: ready for connections.
[--] 9) 180327 3:38:35 [Note] /usr/libexec/mysqld: ready for connections.
[--] 10) 180326 2:33:41 [Note] /usr/libexec/mysqld: ready for connections.
[--] 13 shutdown(s) detected in /var/log/mysqld.log
[--] 1) 180318 5:12:47 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 2) 180310 7:03:41 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 3) 180310 7:01:49 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 4) 180309 4:54:39 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 5) 180309 4:43:33 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 6) 180306 3:43:21 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 7) 180305 6:12:58 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 8) 180305 6:12:27 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 9) 180305 5:51:45 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 10) 180222 15:23:36 [Note] /usr/libexec/mysqld: Shutdown complete

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 237M (Tables: 246)
[--] Data in InnoDB tables: 475M (Tables: 823)
[OK] Total fragmented tables: 0

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] User 'Burundik_bdk@%' hasn't specific host restriction.
[!!] User 'Python_avss@%' hasn't specific host restriction.
[!!] User 'Python_vss@%' hasn't specific host restriction.
[!!] User 'ZemliRUS_1@%' hasn't specific host restriction.
[!!] User 'ZemliRUS_dom@%' hasn't specific host restriction.
[!!] User 'ZemliRUS_fdm@%' hasn't specific host restriction.
[!!] User 'ZemliRUS_test@%' hasn't specific host restriction.
[!!] User 'ZemliRUS_zab@%' hasn't specific host restriction.
[!!] User 'admin_default@%' hasn't specific host restriction.
[!!] User 'domofon-sm.ru_e@%' hasn't specific host restriction.
[!!] User 'eco-pool_1@%' hasn't specific host restriction.
[!!] User 'frezer_1@%' hasn't specific host restriction.
[!!] User 'krovlia_1@%' hasn't specific host restriction.
[!!] User 'python-studio_pm@%' hasn't specific host restriction.
[!!] User 'python-studio_s@%' hasn't specific host restriction.
[!!] User 'sotka.rus_1@%' hasn't specific host restriction.
[!!] User 'svm_bd@%' hasn't specific host restriction.
[!!] User 'vkusbuket_1@%' hasn't specific host restriction.
[!!] User 'yabot_cz@%' hasn't specific host restriction.
[!!] User 'yabot_yabot@%' hasn't specific host restriction.
[!!] There is no basic password file list!

-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 1d 8h 39m 40s (3M q [28.168 qps], 13K conn, TX: 16G, RX: 1G)
[--] Reads / Writes: 44% / 56%
[--] Binary logging is disabled
[--] Physical Memory : 2.0G
[--] Max MySQL memory : 414.9M
[--] Other process memory: 1.9G
[--] Total buffers: 288.0M global + 3.6M per thread (35 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 353.2M (17.25% of installed RAM)
[OK] Maximum possible memory usage: 414.9M (20.26% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (1/3M)
[OK] Highest usage of available connections: 51% (18/35)
[!!] Aborted connections: 7.21% (988/13709)
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 1M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 97K sorts)
[!!] Joins performed without indexes: 3902
[!!] Temporary tables created on disk: 55% (64K on disk / 116K total)
[!!] Thread cache is disabled
[!!] Table cache hit rate: 0% (4 open / 310K opened)
[OK] Open file limit used: 0% (0/1K)
[OK] Table locks acquired immediately: 99% (3M immediate / 3M locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (24M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/90.2M
[!!] Read Key buffer hit rate: 94.9% (51M cached / 2M reads)
[!!] Write Key buffer hit rate: 24.5% (15M cached / 3M writes)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 128.0M/475.8M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (7.8125 %): 5.0M * 2/128.0M should be equal 25%
[OK] InnoDB buffer pool instances: 1
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 99.76% (31928756 hits/ 32004323 total)
[!!] InnoDB Write Log efficiency: 76.07% (732304 hits/ 962611 total)
[OK] InnoDB log waits: 0.00% (0 waits / 230307 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- RocksDB Metrics ---------------------------------------------------------------------------
[--] RocksDB is disabled.

-------- Spider Metrics ----------------------------------------------------------------------------
[--] Spider is disabled.

-------- Connect Metrics ---------------------------------------------------------------------------
[--] Connect is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: STATEMENT
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
Control warning line(s) into /var/log/mysqld.log file
Control error line(s) into /var/log/mysqld.log file
Restrict Host for user@% to user@SpecificDNSorIp
Dedicate this server to your database for highest performance.
Enable the slow query log to troubleshoot bad queries
Reduce or eliminate unclosed connections and network issues
Adjust your join queries to always utilize indexes
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries which have no LIMIT clause
Set thread_cache_size to 4 as a starting value
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64: http://bit.ly/1mi7c4C
This is MyISAM only table_cache scalability problem, InnoDB not affected.
See more details here: https://bugs.mysql.com/bug.php?id=49177
This bug already fixed in MySQL 5.7.9 and newer MySQL versions.
Beware that open_files_limit (1024) variable
should be greater than table_open_cache (4)
Consider installing Sys schema from https://github.com/mysql/mysql-sys
Read this before changing innodb_log_file_size and/or innodb_log_files_in_group: http://bit.ly/2wgkDvS
Variables to adjust:
query_cache_size (=0)
query_cache_type (=0)
query_cache_limit (> 1M, or use smaller result sets)
join_buffer_size (> 128.0K, or always use indexes with joins)
tmp_table_size (> 16M)
max_heap_table_size (> 16M)
thread_cache_size (start at 4)
table_open_cache (> 4)
innodb_buffer_pool_size (>= 475M) if possible.
innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
[root@python-studio ~]#

[ayaroshenko]

На данный момент:
httpd веб сервер
Процессор: 0.9
Память: 2058 Мб
Запущен: 1 дн. (перезапускается раз в день)


mysqld сервер баз данных
Процессор: 0.1
Память: 1907 Мб
Запущен: 1 дн. (перезапускается раз в день)

[ayaroshenko]

Я так понял, что в конфиге нужно прописать:
query_cache_size (=0)
query_cache_type (=0)
query_cache_limit (> 1M, or use smaller result sets)
join_buffer_size (> 128.0K, or always use indexes with joins)
tmp_table_size (> 16M)
max_heap_table_size (> 16M)
thread_cache_size (start at 4)
table_open_cache (> 4)
innodb_buffer_pool_size (>= 475M) if possible.
innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

Вот только вопрос не загнется ли у меня вообще сервак?

Сейчас конфиг выглядит так

SpoilerShow

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
skip-host-cache
skip-name-resolve

skip-external-locking
key_buffer_size = 128M
max_allowed_packet = 1M
table_open_cache = 4
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 1M
net_buffer_length = 2K
thread_stack = 512K

#innodb_use_native_aio = 0
innodb_file_per_table

max_connections=35
max_connect_errors=50
max_user_connections=25
wait_timeout=10
interactive_timeout=50
long_query_time=5

#slow_query_log=1
#slow_query_log_file=/var/log/mysql-slow-queries.log

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

Сервак такой:
OpenVZ SSD VPS (SVZ) - NL - 2048MB SVZ
2048MB RAM
4 CPU Core Access
95GB SSD Space
1Gbps Port
4000GB Bandwidth
1 IPv4 Address
/64 IPv6
TUN/TAP