Vesta Control Panel - Forum

Community Forum
https://forum.vestacp.com/

Database suddenly shutdown

https://forum.vestacp.com/viewtopic.php?t=13315

Page 1 of 1

Database suddenly shutdown

Posted: Mon Dec 05, 2016 4:55 pm
by liquid
I have problem with mysql server. It goes down sometimes, maybe twice every month, idk what's happening. I need to run mysql manually.

This is what I can see in graphs:

Image

And this is log from /var/log/mysqld.log, at time of shutdown:

Code: Select all


161205 19:02:06 mysqld_safe Number of processes running now: 0
161205 19:02:08 mysqld_safe mysqld restarted
161205 19:02:16 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
161205 19:02:16 [Note] /usr/libexec/mysqld (mysqld 5.5.52) starting as process 21376 ...
161205 19:02:34 [Note] Plugin 'FEDERATED' is disabled.
161205 19:02:34 InnoDB: The InnoDB memory heap is disabled
161205 19:02:34 InnoDB: Mutexes and rw_locks use GCC atomic builtins
161205 19:02:34 InnoDB: Compressed tables use zlib 1.2.3
161205 19:02:34 InnoDB: Using Linux native AIO
161205 19:02:34 InnoDB: Initializing buffer pool, size = 128.0M
161205 19:02:34 InnoDB: Completed initialization of buffer pool
161205 19:02:35 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
161205 19:02:35  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
161205 19:02:43  InnoDB: Error: table '*****_****ija/****_wfNet404s'
InnoDB: in InnoDB data dictionary has tablespace id 3637,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:02:43  InnoDB: Error: table '*****_****ija/****_wfVulnScanners'
InnoDB: in InnoDB data dictionary has tablespace id 3638,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:02:43  InnoDB: Waiting for the background threads to start
161205 19:02:44 InnoDB: 5.5.52 started; log sequence number 7512781303
161205 19:02:44 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
161205 19:02:44 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
161205 19:02:44 [Note] Server socket created on IP: '0.0.0.0'.
161205 19:02:45 [Note] Event Scheduler: Loaded 0 events
161205 19:02:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
161205 19:12:20 mysqld_safe Number of processes running now: 0
161205 19:12:20 mysqld_safe mysqld restarted
161205 19:12:23 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
161205 19:12:23 [Note] /usr/libexec/mysqld (mysqld 5.5.52) starting as process 22534 ...
161205 19:12:24 [Note] Plugin 'FEDERATED' is disabled.
161205 19:12:24 InnoDB: The InnoDB memory heap is disabled
161205 19:12:24 InnoDB: Mutexes and rw_locks use GCC atomic builtins
161205 19:12:24 InnoDB: Compressed tables use zlib 1.2.3
161205 19:12:24 InnoDB: Using Linux native AIO
161205 19:12:24 InnoDB: Initializing buffer pool, size = 128.0M
161205 19:12:24 InnoDB: Completed initialization of buffer pool
161205 19:12:24 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
161205 19:12:24  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
161205 19:12:30  InnoDB: Error: table '*****_****ija/****_wfNet404s'
InnoDB: in InnoDB data dictionary has tablespace id 3637,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:12:30  InnoDB: Error: table '*****_****ija/****_wfVulnScanners'
InnoDB: in InnoDB data dictionary has tablespace id 3638,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:12:30  InnoDB: Waiting for the background threads to start
161205 19:12:31 InnoDB: 5.5.52 started; log sequence number 7513101031
161205 19:12:31 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
161205 19:12:31 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
161205 19:12:31 [Note] Server socket created on IP: '0.0.0.0'.
161205 19:12:32 [Note] Event Scheduler: Loaded 0 events
161205 19:12:32 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
161205 19:15:17 mysqld_safe Number of processes running now: 0
161205 19:15:17 mysqld_safe mysqld restarted
161205 19:15:18 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
161205 19:15:18 [Note] /usr/libexec/mysqld (mysqld 5.5.52) starting as process 22924 ...
161205 19:15:18 [Note] Plugin 'FEDERATED' is disabled.
161205 19:15:18 InnoDB: The InnoDB memory heap is disabled
161205 19:15:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
161205 19:15:18 InnoDB: Compressed tables use zlib 1.2.3
161205 19:15:18 InnoDB: Using Linux native AIO
161205 19:15:18 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
161205 19:15:18 InnoDB: Completed initialization of buffer pool
161205 19:15:18 InnoDB: Fatal error: cannot allocate memory for the buffer pool
161205 19:15:18 [ERROR] Plugin 'InnoDB' init function returned error.
161205 19:15:18 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
161205 19:15:18 [ERROR] Unknown/unsupported storage engine: InnoDB
161205 19:15:18 [ERROR] Aborting

161205 19:15:18 [Note] /usr/libexec/mysqld: Shutdown complete

161205 19:15:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
161205 19:26:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
161205 19:26:53 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
161205 19:26:53 [Note] /usr/libexec/mysqld (mysqld 5.5.52) starting as process 24179 ...
161205 19:26:53 [Note] Plugin 'FEDERATED' is disabled.
161205 19:26:53 InnoDB: The InnoDB memory heap is disabled
161205 19:26:53 InnoDB: Mutexes and rw_locks use GCC atomic builtins
161205 19:26:53 InnoDB: Compressed tables use zlib 1.2.3
161205 19:26:53 InnoDB: Using Linux native AIO
161205 19:26:53 InnoDB: Initializing buffer pool, size = 128.0M
161205 19:26:53 InnoDB: Completed initialization of buffer pool
161205 19:26:53 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
161205 19:26:53  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
161205 19:26:57  InnoDB: Error: table '*****_****ija/****_wfNet404s'
InnoDB: in InnoDB data dictionary has tablespace id 3637,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:26:57  InnoDB: Error: table '*****_****ija/****_wfVulnScanners'
InnoDB: in InnoDB data dictionary has tablespace id 3638,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:26:57  InnoDB: Waiting for the background threads to start
161205 19:26:58 InnoDB: 5.5.52 started; log sequence number 7513152114
161205 19:26:58 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
161205 19:26:58 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
161205 19:26:58 [Note] Server socket created on IP: '0.0.0.0'.
161205 19:26:59 [Note] Event Scheduler: Loaded 0 events
161205 19:26:59 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
I can see that there is some problem with tables wfNet404s and wfVulnScanners ... that's tabels of wordfence plugin, but I removed that plugin and that tables ...

I don't even know what to google, this log is confusing :S I suppose that there is some problem with overload of something, is there any way to limit somethink ... I'm really confused

Thank you!

Re: Database suddenly shutdown

Posted: Mon Dec 05, 2016 7:16 pm
by liquid
UPDATE: I think that was some kind of attack, I found IPs via iftop that used huge % of server so I droped that IPs with iptables and seems that problem is resolved, sorry for topic :(

Image

Re: Database suddenly shutdown

Posted: Mon Dec 05, 2016 8:47 pm
by skamasle
Hi

Can be attack yes

And about wfNet404s and wfVulnScanners

This is a problem whit wordfence you will need fix that tables manually because it never delete it in good way

The main problems whit that tables is than you may be affected in backup

Try a backup your database whit mysqldump in most cases you cant do a backups if you have broken that both tables.
All times are UTC
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited