We are happy to announce that Vesta is back under active development as of 25 February 2024. We are working on v1 candidate and expect to engage more with the community over the coming months. We are committed to open source, and we encourage contributors to help us build the future of Vesta.
Database suddenly shutdown
Database suddenly shutdown
I have problem with mysql server. It goes down sometimes, maybe twice every month, idk what's happening. I need to run mysql manually.
This is what I can see in graphs:
And this is log from /var/log/mysqld.log, at time of shutdown:
I can see that there is some problem with tables wfNet404s and wfVulnScanners ... that's tabels of wordfence plugin, but I removed that plugin and that tables ...
I don't even know what to google, this log is confusing :S I suppose that there is some problem with overload of something, is there any way to limit somethink ... I'm really confused
Thank you!
This is what I can see in graphs:
And this is log from /var/log/mysqld.log, at time of shutdown:
Code: Select all
161205 19:02:06 mysqld_safe Number of processes running now: 0
161205 19:02:08 mysqld_safe mysqld restarted
161205 19:02:16 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
161205 19:02:16 [Note] /usr/libexec/mysqld (mysqld 5.5.52) starting as process 21376 ...
161205 19:02:34 [Note] Plugin 'FEDERATED' is disabled.
161205 19:02:34 InnoDB: The InnoDB memory heap is disabled
161205 19:02:34 InnoDB: Mutexes and rw_locks use GCC atomic builtins
161205 19:02:34 InnoDB: Compressed tables use zlib 1.2.3
161205 19:02:34 InnoDB: Using Linux native AIO
161205 19:02:34 InnoDB: Initializing buffer pool, size = 128.0M
161205 19:02:34 InnoDB: Completed initialization of buffer pool
161205 19:02:35 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
161205 19:02:35 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
161205 19:02:43 InnoDB: Error: table '*****_****ija/****_wfNet404s'
InnoDB: in InnoDB data dictionary has tablespace id 3637,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:02:43 InnoDB: Error: table '*****_****ija/****_wfVulnScanners'
InnoDB: in InnoDB data dictionary has tablespace id 3638,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:02:43 InnoDB: Waiting for the background threads to start
161205 19:02:44 InnoDB: 5.5.52 started; log sequence number 7512781303
161205 19:02:44 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
161205 19:02:44 [Note] - '0.0.0.0' resolves to '0.0.0.0';
161205 19:02:44 [Note] Server socket created on IP: '0.0.0.0'.
161205 19:02:45 [Note] Event Scheduler: Loaded 0 events
161205 19:02:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
161205 19:12:20 mysqld_safe Number of processes running now: 0
161205 19:12:20 mysqld_safe mysqld restarted
161205 19:12:23 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
161205 19:12:23 [Note] /usr/libexec/mysqld (mysqld 5.5.52) starting as process 22534 ...
161205 19:12:24 [Note] Plugin 'FEDERATED' is disabled.
161205 19:12:24 InnoDB: The InnoDB memory heap is disabled
161205 19:12:24 InnoDB: Mutexes and rw_locks use GCC atomic builtins
161205 19:12:24 InnoDB: Compressed tables use zlib 1.2.3
161205 19:12:24 InnoDB: Using Linux native AIO
161205 19:12:24 InnoDB: Initializing buffer pool, size = 128.0M
161205 19:12:24 InnoDB: Completed initialization of buffer pool
161205 19:12:24 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
161205 19:12:24 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
161205 19:12:30 InnoDB: Error: table '*****_****ija/****_wfNet404s'
InnoDB: in InnoDB data dictionary has tablespace id 3637,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:12:30 InnoDB: Error: table '*****_****ija/****_wfVulnScanners'
InnoDB: in InnoDB data dictionary has tablespace id 3638,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:12:30 InnoDB: Waiting for the background threads to start
161205 19:12:31 InnoDB: 5.5.52 started; log sequence number 7513101031
161205 19:12:31 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
161205 19:12:31 [Note] - '0.0.0.0' resolves to '0.0.0.0';
161205 19:12:31 [Note] Server socket created on IP: '0.0.0.0'.
161205 19:12:32 [Note] Event Scheduler: Loaded 0 events
161205 19:12:32 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
161205 19:15:17 mysqld_safe Number of processes running now: 0
161205 19:15:17 mysqld_safe mysqld restarted
161205 19:15:18 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
161205 19:15:18 [Note] /usr/libexec/mysqld (mysqld 5.5.52) starting as process 22924 ...
161205 19:15:18 [Note] Plugin 'FEDERATED' is disabled.
161205 19:15:18 InnoDB: The InnoDB memory heap is disabled
161205 19:15:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
161205 19:15:18 InnoDB: Compressed tables use zlib 1.2.3
161205 19:15:18 InnoDB: Using Linux native AIO
161205 19:15:18 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
161205 19:15:18 InnoDB: Completed initialization of buffer pool
161205 19:15:18 InnoDB: Fatal error: cannot allocate memory for the buffer pool
161205 19:15:18 [ERROR] Plugin 'InnoDB' init function returned error.
161205 19:15:18 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
161205 19:15:18 [ERROR] Unknown/unsupported storage engine: InnoDB
161205 19:15:18 [ERROR] Aborting
161205 19:15:18 [Note] /usr/libexec/mysqld: Shutdown complete
161205 19:15:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
161205 19:26:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
161205 19:26:53 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
161205 19:26:53 [Note] /usr/libexec/mysqld (mysqld 5.5.52) starting as process 24179 ...
161205 19:26:53 [Note] Plugin 'FEDERATED' is disabled.
161205 19:26:53 InnoDB: The InnoDB memory heap is disabled
161205 19:26:53 InnoDB: Mutexes and rw_locks use GCC atomic builtins
161205 19:26:53 InnoDB: Compressed tables use zlib 1.2.3
161205 19:26:53 InnoDB: Using Linux native AIO
161205 19:26:53 InnoDB: Initializing buffer pool, size = 128.0M
161205 19:26:53 InnoDB: Completed initialization of buffer pool
161205 19:26:53 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
161205 19:26:53 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
161205 19:26:57 InnoDB: Error: table '*****_****ija/****_wfNet404s'
InnoDB: in InnoDB data dictionary has tablespace id 3637,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:26:57 InnoDB: Error: table '*****_****ija/****_wfVulnScanners'
InnoDB: in InnoDB data dictionary has tablespace id 3638,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
161205 19:26:57 InnoDB: Waiting for the background threads to start
161205 19:26:58 InnoDB: 5.5.52 started; log sequence number 7513152114
161205 19:26:58 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
161205 19:26:58 [Note] - '0.0.0.0' resolves to '0.0.0.0';
161205 19:26:58 [Note] Server socket created on IP: '0.0.0.0'.
161205 19:26:59 [Note] Event Scheduler: Loaded 0 events
161205 19:26:59 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
I don't even know what to google, this log is confusing :S I suppose that there is some problem with overload of something, is there any way to limit somethink ... I'm really confused
Thank you!
Re: Database suddenly shutdown
UPDATE: I think that was some kind of attack, I found IPs via iftop that used huge % of server so I droped that IPs with iptables and seems that problem is resolved, sorry for topic :(
Re: Database suddenly shutdown
Hi
Can be attack yes
And about wfNet404s and wfVulnScanners
This is a problem whit wordfence you will need fix that tables manually because it never delete it in good way
The main problems whit that tables is than you may be affected in backup
Try a backup your database whit mysqldump in most cases you cant do a backups if you have broken that both tables.
Can be attack yes
And about wfNet404s and wfVulnScanners
This is a problem whit wordfence you will need fix that tables manually because it never delete it in good way
The main problems whit that tables is than you may be affected in backup
Try a backup your database whit mysqldump in most cases you cant do a backups if you have broken that both tables.