Постоянно падает mysql Centos 6.7, куда копать?

Question

[Denis Denisov]

День добрый!

Постоянно падает база и уходит в глухой даун. Помогает только переустановка всего)
Centos 6.7

[root@ns398679 ~]# rpm -qa | grep mysql
mysql-5.5.48-1.el6.remi.x86_64
compat-mysql51-5.1.54-1.el6.remi.x86_64
mysql-libs-5.5.48-1.el6.remi.x86_64
php-mysql-5.4.45-4.el6.remi.x86_64
mysql-server-5.5.48-1.el6.remi.x86_64

14:58:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 15:04:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160223 15:04:20 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
160223 15:04:20 [Note] /usr/libexec/mysqld (mysqld 5.5.48) starting as process 14267 ...
160223 15:04:20 [Note] Plugin 'FEDERATED' is disabled.
160223 15:04:20 InnoDB: The InnoDB memory heap is disabled
160223 15:04:20 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 15:04:20 InnoDB: Compressed tables use zlib 1.2.3
160223 15:04:20 InnoDB: Using Linux native AIO
160223 15:04:20 InnoDB: Initializing buffer pool, size = 128.0M
160223 15:04:20 InnoDB: Completed initialization of buffer pool
160223 15:04:20 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 19182772868
160223 15:04:20 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 1325 of name './admin_immigrat/wp_termmeta.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 1325 of name './immigration/wp_termmeta.ibd' already exists in the tablespace
InnoDB: memory cache!
160223 15:04:20 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 15:04:26 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160223 15:04:26 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
160223 15:04:26 [Note] /usr/libexec/mysqld (mysqld 5.5.48) starting as process 14964 ...
160223 15:04:26 [Note] Plugin 'FEDERATED' is disabled.
160223 15:04:26 InnoDB: The InnoDB memory heap is disabled
160223 15:04:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 15:04:26 InnoDB: Compressed tables use zlib 1.2.3
160223 15:04:26 InnoDB: Using Linux native AIO
160223 15:04:26 InnoDB: Initializing buffer pool, size = 128.0M
160223 15:04:26 InnoDB: Completed initialization of buffer pool
160223 15:04:26 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 19182772868
160223 15:04:26 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 1325 of name './admin_immigrat/wp_termmeta.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 1325 of name './immigration/wp_termmeta.ibd' already exists in the tablespace
noDB: memory cache!
160223 15:04:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 15:05:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160223 15:05:52 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
160223 15:05:52 [Note] /usr/libexec/mysqld (mysqld 5.5.48) starting as process 1684 ...
160223 15:05:52 [Note] Plugin 'FEDERATED' is disabled.
160223 15:05:52 InnoDB: The InnoDB memory heap is disabled
160223 15:05:52 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 15:05:52 InnoDB: Compressed tables use zlib 1.2.3
160223 15:05:52 InnoDB: Using Linux native AIO
160223 15:05:52 InnoDB: Initializing buffer pool, size = 128.0M
160223 15:05:52 InnoDB: Completed initialization of buffer pool
160223 15:05:52 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 19182772868
160223 15:05:52 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 1325 of name './admin_immigrat/wp_termmeta.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 1325 of name './immigration/wp_termmeta.ibd' already exists in the tablespace
InnoDB: memory cache!
160223 15:05:52 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 15:14:31 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160223 15:14:31 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
160223 15:14:31 [Note] /usr/libexec/mysqld (mysqld 5.5.48) starting as process 1683 ...
160223 15:14:31 [Note] Plugin 'FEDERATED' is disabled.
160223 15:14:31 InnoDB: The InnoDB memory heap is disabled
160223 15:14:31 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 15:14:31 InnoDB: Compressed tables use zlib 1.2.3
160223 15:14:31 InnoDB: Using Linux native AIO
160223 15:4:31 InnoDB: Initializing buffer pool, size = 128.0M
160223 15:14:31 InnoDB: Completed initialization of buffer pool
160223 15:14:31 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 19182772868
160223 15:14:31 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 1325 of name './admin_immigrat/wp_termmeta.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 1325 of name './immigration/wp_termmeta.ibd' already exists in the tablespace
InnoDB: memory cache!
160223 15:14:31 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 15:25:26 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160223 15:25:26 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
160223 15:25:26 [Note] /usr/libexec/mysqld (mysqld 5.5.48) starting as process 1683 ...
160223 15:25:26 [Note] Plugin 'FEDERATED' is disabled.
160223 15:25:26 InnoDB: The InnoDB memory heap is disabled
160223 15:25:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 15:25:26 InnoDB: Compressed tables use zlib 1.2.3
160223 15:25:26 InnoDB: Using Linux native AIO
160223 15:25:26 InnoDB: Initializing buffer pool, size = 128.0M
160223 15:25:26 InnoDB: Completed initialization of buffer pool
160223 15:25:26 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 19182772868
160223 15:25:26 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 1325 of name './admin_immigrat/wp_termmeta.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 1325 of name './immigration/wp_termmeta.ibd' already exists in the tablespace
InnoDB: memory cache!
160223 15:25:26 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Answer 1

[Игорь]

Места ему везде хватает?
Оперативной памяти на всякие буферы?
юзается ли swap?

Comments

[Игорь]

и посмотрите еще системный лог, может там что подробнее есть

[Denis Denisov]

оперативы 32 Гб, swap да юзается. Изначально думал как раз из-за нехватке места переустановил ось - 20Гб свободного места, а все равно падает

[Denis Denisov]

160223 18:38:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 18:38:15 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 6138 ...
160223 18:38:15 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 18:38:15 [Note] Plugin 'FEDERATED' is disabled.
160223 18:38:15 InnoDB: The InnoDB memory heap is disabled
160223 18:38:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 18:38:15 InnoDB: Compressed tables use zlib 1.2.3
160223 18:38:15 InnoDB: Using Linux native AIO
160223 18:38:15 InnoDB: Initializing buffer pool, size = 26.0G
160223 18:38:16 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 5242880 bytes
InnoDB: than specified in the .cnf file 0 536870912 bytes!
160223 18:38:16 [ERROR] Plugin 'InnoDB' init function returned error.
160223 18:38:16 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160223 18:38:16 [ERROR] Failed to initialize plugins.
160223 18:38:16 [ERROR] Aborting

160223 18:38:16 [Note] /usr/libexec/mysqld: Shutdown complete

160223 18:38:17 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:11:47 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:11:47 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 11725 ...
160223 19:11:47 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:11:47 [Note] Plugin 'FEDERATED' is disabled.
160223 19:11:47 InnoDB: The InnoDB memory heap is disabled
160223 19:11:47 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:11:47 InnoDB: Compressed tables use zlib 1.2.3
160223 19:11:47 InnoDB: Using Linux native AIO
160223 19:11:47 InnoDB: Initializing buffer pool, size = 26.0G
160223 19:11:48 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 5242880 bytes
InnoDB: than specified in the .cnf file 0 536870912 bytes!
160223 19:11:48 [ERROR] Plugin 'InnoDB' init function returned error.
160223 19:11:48 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160223 19:11:48 [ERROR] Failed to initialize plugins.
160223 19:11:48 [ERROR] Aborting

160223 19:11:48 [Note] /usr/libexec/mysqld: Shutdown complete

160223 19:11:49 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:19:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:19:18 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 13086 ...
160223 19:19:18 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:19:18 [Note] Plugin 'FEDERATED' is disabled.
160223 19:19:18 InnoDB: The InnoDB memory heap is disabled
160223 19:19:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:19:18 InnoDB: Compressed tables use zlib 1.2.3
160223 19:19:18 InnoDB: Using Linux native AIO
160223 19:19:18 InnoDB: Initializing buffer pool, size = 128.0M
160223 19:19:18 InnoDB: Completed initialization of buffer pool
160223 19:19:18 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 19182772868
160223 19:19:18 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 1325 of name './admin_immigrat/wp_termmeta.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 1325 of name './immigration/wp_termmeta.ibd' already exists in the tablespace
InnoDB: memory cache!
160223 19:19:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:19:23 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:19:23 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 13637 ...
160223 19:19:23 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:19:23 [Note] Plugin 'FEDERATED' is disabled.
160223 19:19:23 InnoDB: The InnoDB memory heap is disabled
160223 19:19:23 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:19:23 InnoDB: Compressed tables use zlib 1.2.3
160223 19:19:23 InnoDB: Using Linux native AIO
160223 19:19:23 InnoDB: Initializing buffer pool, size = 128.0M
160223 19:19:23 InnoDB: Completed initialization of buffer pool
160223 19:19:23 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 19182772868
160223 19:19:23 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 1325 of name './admin_immigrat/wp_termmeta.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 1325 of name './immigration/wp_termmeta.ibd' already exists in the tablespace
InnoDB: memory cache!
160223 19:19:23 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:41:33 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:41:33 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 17372 ...
160223 19:41:33 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:41:33 [Note] Plugin 'FEDERATED' is disabled.
160223 19:41:33 InnoDB: The InnoDB memory heap is disabled
160223 19:41:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:41:33 InnoDB: Compressed tables use zlib 1.2.3
160223 19:41:33 InnoDB: Using Linux native AIO
160223 19:41:33 InnoDB: Initializing buffer pool, size = 26.0G
160223 19:41:33 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 5242880 bytes
InnoDB: than specified in the .cnf file 0 536870912 bytes!
160223 19:41:33 [ERROR] Plugin 'InnoDB' init function returned error.
160223 19:41:33 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160223 19:41:33 [ERROR] Failed to initialize plugins.
160223 19:41:33 [ERROR] Aborting

160223 19:41:33 [Note] /usr/libexec/mysqld: Shutdown complete

160223 19:41:34 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:42:26 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:42:26 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 18064 ...
160223 19:42:26 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:42:26 [Note] Plugin 'FEDERATED' is disabled.
160223 19:42:26 InnoDB: The InnoDB memory heap is disabled
160223 19:42:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:42:26 InnoDB: Compressed tables use zlib 1.2.3
160223 19:42:26 InnoDB: Using Linux native AIO
160223 19:42:26 InnoDB: Initializing buffer pool, size = 26.0G
160223 19:42:27 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 5242880 bytes
InnoDB: than specified in the .cnf file 0 536870912 bytes!
160223 19:42:27 [ERROR] Plugin 'InnoDB' init function returned error.
160223 19:42:27 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160223 19:42:27 [ERROR] Failed to initialize plugins.
160223 19:42:27 [ERROR] Aborting

160223 19:42:27 [Note] /usr/libexec/mysqld: Shutdown complete

160223 19:42:28 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:42:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:42:59 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 18725 ...
160223 19:42:59 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:42:59 [Note] Plugin 'FEDERATED' is disabled.
160223 19:42:59 InnoDB: The InnoDB memory heap is disabled
160223 19:42:59 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:42:59 InnoDB: Compressed tables use zlib 1.2.3
160223 19:42:59 InnoDB: Using Linux native AIO
160223 19:42:59 InnoDB: Initializing buffer pool, size = 26.0G
160223 19:43:00 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 5242880 bytes
InnoDB: than specified in the .cnf file 0 536870912 bytes!
160223 19:43:00 [ERROR] Plugin 'InnoDB' init function returned error.
160223 19:43:00 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160223 19:43:00 [ERROR] Failed to initialize plugins.
160223 19:43:00 [ERROR] Aborting

160223 19:43:00 [Note] /usr/libexec/mysqld: Shutdown complete

160223 19:43:00 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:46:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:46:14 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 19955 ...
160223 19:46:14 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:46:14 [Note] Plugin 'FEDERATED' is disabled.
160223 19:46:14 InnoDB: The InnoDB memory heap is disabled
160223 19:46:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:46:14 InnoDB: Compressed tables use zlib 1.2.3
160223 19:46:14 InnoDB: Using Linux native AIO
160223 19:46:14 InnoDB: Initializing buffer pool, size = 26.0G
160223 19:46:14 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 5242880 bytes
InnoDB: than specified in the .cnf file 0 536870912 bytes!
160223 19:46:14 [ERROR] Plugin 'InnoDB' init function returned error.
160223 19:46:14 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160223 19:46:14 [ERROR] Failed to initialize plugins.
160223 19:46:14 [ERROR] Aborting

160223 19:46:14 [Note] /usr/libexec/mysqld: Shutdown complete

160223 19:46:15 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:46:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:46:58 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 20642 ...
160223 19:46:58 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:46:58 [Note] Plugin 'FEDERATED' is disabled.
160223 19:46:58 InnoDB: The InnoDB memory heap is disabled
160223 19:46:58 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:46:58 InnoDB: Compressed tables use zlib 1.2.3
160223 19:46:58 InnoDB: Using Linux native AIO
160223 19:46:58 InnoDB: Initializing buffer pool, size = 26.0G
160223 19:46:59 InnoDB: Completed initialization of buffer pool
160223 19:46:59 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 512 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100 200 300 400 500
160223 19:47:04 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 512 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100 200 300 400 500
160223 19:47:08 InnoDB: highest supported file format is Barracuda.
InnoDB: The user has set SRV_FORCE_NO_LOG_REDO on
InnoDB: Skipping log redo
160223 19:47:09 InnoDB: Waiting for the background threads to start
160223 19:47:10 InnoDB: 5.5.48 started; log sequence number 0
160223 19:47:10 InnoDB: !!! innodb_force_recovery is set to 6 !!!
160223 19:47:10 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
160223 19:47:10 [Note] - '0.0.0.0' resolves to '0.0.0.0';
160223 19:47:10 [Note] Server socket created on IP: '0.0.0.0'.
160223 19:47:10 [ERROR] /usr/libexec/mysqld: Table './mysql/user' is marked as crashed and should be repaired
160223 19:47:10 [Warning] Checking table: './mysql/user'
160223 19:47:10 [ERROR] 1 client is using or hasn't closed the table properly
160223 19:47:10 [ERROR] /usr/libexec/mysqld: Table './mysql/db' is marked as crashed and should be repaired
160223 19:47:10 [Warning] Checking table: './mysql/db'
160223 19:47:10 [ERROR] 1 client is using or hasn't closed the table properly
160223 19:47:10 [Warning] 'proxies_priv' entry '@ [e-mail]' ignored in --skip-name-resolve mode.
160223 19:47:10 [Note] Event Scheduler: Loaded 0 events
160223 19:47:10 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.48-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
160223 19:48:59 [Note] /usr/libexec/mysqld: Normal shutdown

160223 19:48:59 [Note] Event Scheduler: Purging the queue. 0 events
160223 19:48:59 InnoDB: Starting shutdown...
160223 19:49:00 InnoDB: Shutdown completed; log sequence number 9234869772
160223 19:49:00 [Note] /usr/libexec/mysqld: Shutdown complete

160223 19:49:01 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:49:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:49:01 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 21519 ...
160223 19:49:01 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:49:01 [Note] Plugin 'FEDERATED' is disabled.
160223 19:49:01 InnoDB: The InnoDB memory heap is disabled
160223 19:49:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:49:01 InnoDB: Compressed tables use zlib 1.2.3
160223 19:49:01 InnoDB: Using Linux native AIO
160223 19:49:01 InnoDB: Initializing buffer pool, size = 26.0G
160223 19:49:02 InnoDB: Completed initialization of buffer pool
160223 19:49:02 InnoDB: highest supported file format is Barracuda.
InnoDB: The user has set SRV_FORCE_NO_LOG_REDO on
InnoDB: Skipping log redo
160223 19:49:02 InnoDB: Waiting for the background threads to start
160223 19:49:03 InnoDB: 5.5.48 started; log sequence number 0
160223 19:49:03 InnoDB: !!! innodb_force_recovery is set to 6 !!!
160223 19:49:03 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
160223 19:49:03 [Note] - '0.0.0.0' resolves to '0.0.0.0';
160223 19:49:03 [Note] Server socket created on IP: '0.0.0.0'.
160223 19:49:03 [Warning] 'proxies_priv' entry '@ [e-mail]' ignored in --skip-name-resolve mode.
160223 19:49:03 [Note] Event Scheduler: Loaded 0 events
160223 19:49:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.48-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
160223 19:50:50 [Note] /usr/libexec/mysqld: Normal shutdown

160223 19:50:50 [Note] Event Scheduler: Purging the queue. 0 events
160223 19:50:50 InnoDB: Starting shutdown...
160223 19:50:51 InnoDB: Shutdown completed; log sequence number 8204
160223 19:50:51 [Note] /usr/libexec/mysqld: Shutdown complete

160223 19:50:52 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160223 19:50:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
160223 19:50:52 [Note] /usr/libexec/mysqld (mysqld 5.5.48-log) starting as process 22677 ...
160223 19:50:52 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160223 19:50:52 [Note] Plugin 'FEDERATED' is disabled.
160223 19:50:52 InnoDB: The InnoDB memory heap is disabled
160223 19:50:52 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160223 19:50:52 InnoDB: Compressed tables use zlib 1.2.3
160223 19:50:52 InnoDB: Using Linux native AIO
160223 19:50:52 InnoDB: Initializing buffer pool, size = 26.0G
160223 19:50:53 InnoDB: Completed initialization of buffer pool
160223 19:50:53 InnoDB: highest supported file format is Barracuda.
160223 19:50:53 InnoDB: Error: page 7 log sequence number 19180895922
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...
InnoDB: for more information.
160223 19:50:53 InnoDB: Error: page 1 log sequence number 19180898618
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...
InnoDB: for more information.
160223 19:50:53 InnoDB: Error: page 3 log sequence number 6847305491
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...
InnoDB: for more information.
160223 19:50:53 InnoDB: Error: page 2 log sequence number 19180106206
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...
InnoDB: for more information.
160223 19:50:53 InnoDB: Error: page 4 log sequence number 19182211429
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...
InnoDB: for more information.
160223 19:50:53 InnoDB: Error: page 11 log sequence number 19180106206
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...
InnoDB: for more information.
160223 19:50:53 InnoDB: Error: page 349 log sequence number 15290664222
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...
InnoDB: for more information.
160223 19:50:53 InnoDB: Error: page 5 log sequence number 19182772491
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB:

[Игорь]

Denis: не могу точно сказать, но он у вас вначале ругался на настройки innodb и реальный файл. в потом еще и типо сам по себе в shutdown по команде уходит (возможно изза ошибок).
вы переустановили и подсунули старые данные или это ситуация с полной чистой переустановкой без старых даных?

[Denis Denisov]

Игорь: вы переустановили и подсунули старые данные

[Denis Denisov]

Игорь: не все бд, а парочку

[Игорь]

Denis: если не подсовывать ситуация такаяже?

[Denis Denisov]

Игорь: да. Думаю или снести все и с 0 восстанавливать БД через mysql -uroot -p -D имя_базы < имя_базы.sql или поудалять все бд (порядка 36 - 3 из них 2-3Гб) и заново залить

[Игорь]

Denis: я бы посоветовал снести все базы и возращать по одной, может сможете понять при какой падает. плюс начать с тех что не innodb. если судить по логам и файликмо что есть, то вместо записи в таблицу оно крашиться при какомто запросе insert. (именно innodb )

[Игорь]

Denis: и на всякий случай, ето не облако? ну сервак не в облаке?

[Denis Denisov]

Игорь: нет это физический сервак

[Игорь]

Denis: тогда пробуйте. и если есть возможность винт проверьте

[Игорь]

Denis: или хотябы smart гляньте. там же не ssd?

[Denis Denisov]

Игорь: именно ssd

[Denis Denisov]

+soft RAID

[Denis Denisov]

Игорь: ни когда не проверял (лол) подскажите как?

[Игорь]

Denis: тогда ой. :( смотрите S.M.A.R.T. и параметр переназначенных секторов. он у вас видать посыпался. вот и падает ибо не может записать. попробую погуглить как ведет себя innodb в условиях нестабильного винта

[Игорь]

Denis:
www.stableit.ru/2009/10/smart-centos.html
youngblog.hoster-ok.com/obnovlenie-bazy-diskov-sma...
reallocated sector count вот ето цикавит

[Denis Denisov]

cat /proc/mdstat

md1 : active raid1 sdb1[1] sdc1[2] sda1[0]
51198912 blocks [3/3] [UUU]

md2 : active raid1 sdb2[1] sdc2[2] sda2[0]
65491904 blocks [3/3] [UUU]

кажется одно диска не хватает? должно быть 3

[Игорь]

Denis: не знаю, ваш сервер :) для зеркала (raid 1) два нужно. а вот разные размеры меня уже пугают

Answer 2

[CityCat4]

/etc/mysql.cnf (или какой там у Вас конфиг) покажите, если он есть

Comments

[Denis Denisov]

# Example MySQL config file for large systems.
#
# This is for a large system with memory = 512M where the system runs mainly
# MySQL.
#
# MySQL programs look for option files in a set of
# locations which depend on the deployment platform.
# You can copy this option file to one of those
# locations. For information about these locations, see:
# dev.mysql.com/doc/mysql/en/option-files.html
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.

# The following options will be passed to all MySQL clients
[client]
#password = your_password
port = 3306
socket = /var/lib/mysql/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-external-locking
#key_buffer_size = 256M
#max_allowed_packet = 1M
#table_open_cache = 256
#sort_buffer_size = 1M
#read_buffer_size = 1M
#read_rnd_buffer_size = 4M
#myisam_sort_buffer_size = 64M
#thread_cache_size = 8
#query_cache_size= 16M

key_buffer_size = 2560M
max_allowed_packet = 256M
table_open_cache = 25600
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 256M
thread_cache_size = 32
query_cache_size= 256M

# Try number of CPU's*2 for thread_concurrency
#8
thread_concurrency = 12

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin

# binary logging format - mixed recommended
binlog_format=mixed

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=, MASTER_PORT=,
# MASTER_USER=, MASTER_PASSWORD= ;
#
# where you replace , , by quoted strings and
# by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host =
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user =
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port =
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /var/lib/mysql
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /var/lib/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 256M
#innodb_additional_mem_pool_size = 32M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 64M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 2
#innodb_lock_wait_timeout = 50

[mysqldump]
quick
#max_allowed_packet = 16M

max_allowed_packet = 256M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

[Denis Denisov]

было, а вот с помощью перконы

# Generated by Percona Configuration Wizard (tools.percona.com/) version REL5-20120208
# Configuration name server generated for ... at 2016-02-23 17:35:58

[mysql]

# CLIENT #
port = 3306
socket = /var/lib/mysql/mysql.sock

[mysqld]
#innodb_force_recovery=6
# GENERAL #
user = mysql
#default-storage-engine = InnoDB
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid

# MyISAM #
key-buffer-size = 32M
myisam-recover = FORCE,BACKUP

# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
skip-name-resolve
sql-mode = STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
sysdate-is-now = 1
innodb = FORCE
innodb-strict-mode = 1

# DATA STORAGE #
datadir = /var/lib/mysql/

# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1

# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 50
open-files-limit = 65535
table-definition-cache = 4096
table-open-cache = 10240

# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 512M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 26G

# LOGGING #
log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 1
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log

Answer 3

[Дмитрий]

Посмотрите, например, не увеличился ли ibdata file до максимальных значений.

Comments

[Denis Denisov]

3074M видимо в этом и причина, а как увеличить? на сервере 32ГБ

[Дмитрий]

Denis: dev.mysql.com/doc/refman/5.7/en/innodb-configurati... за размер ibdata file отвечает директива innodb_data_file_path вы можете попробовать увеличить максимальное значение размера файла, указав ее в my.cnf, но размер файла ни о чем не говорит, он может быть и 30720М.
Посмотрите в логи внимательно, у вас также сыпятся ошибки для бд, причина может быть и в них, и их необходимо привести к более рабочему состоянию или восстановить из резервных копий.

[Denis Denisov]

InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...
InnoDB: for more information.
160223 19:50:56 InnoDB: Error: page 244 log sequence number 19177288581
InnoDB: is in the future! Current system log sequence number 411184.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-rec...

[Denis Denisov]

Denis: mysql-error.log

[Denis Denisov]

Дмитрий: беда в том, что это самые свежие бэкапы БД

[Denis Denisov]

вот ниже скрин -bin.00000 файлы появляются

Answer 4

[Denis Denisov]

вот какая то хрень появляется и как Вы говорили Дмитрий файл увеличивается как то уж быстро - и все error

Comments

[Игорь]

Посмотрите что в файле с error (f3 в mc)

[Denis Denisov]

Игорь: 38 M что именно смотреть?

[Игорь]

Denis: содержимое

[Denis Denisov]

выложил