Почему падает MySQL?

Question

[neonox]

Всем доброго времени суток. Прошу помочь разобраться в проблеме. Есть виртуальная машина Битрикс 7.3.0 с 5.7.21-21 Percona Server. По какой то причине MySQL начал падать при бекапе.
Что пробовал сделать:
1. Запуск mysqldump из консоли с записью в лог. В логе получаю запись

mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table `b_crm_act` at row: 3848

2. Попробовал создать таблицу b_crm_act_new потом insert into b_crm_act_new select * from b_crm_act, mysql тоже падает. При попытке сделать select * from b_crm_act;
Получаю ERROR 2013 (HY000): Lost connection to MySQL server during query
В логе следующая запись

error.log

19:55:47 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=50331648
read_buffer_size=131072
max_used_connections=5
max_threads=44
thread_count=1
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 407533 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f25880bd340
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f25bff9dd30 thread_stack 0x80000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xef5c1b]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x79b061]
/lib64/libpthread.so.0(+0xf6d0)[0x7f261e1e56d0]
/lib64/libc.so.6(gsignal+0x37)[0x7f261c2e6277]
/lib64/libc.so.6(abort+0x148)[0x7f261c2e7968]
/usr/sbin/mysqld[0x7687ec]
/usr/sbin/mysqld[0x11994ff]
/usr/sbin/mysqld[0x11a9055]
/usr/sbin/mysqld(_Z32btr_copy_externally_stored_fieldPmPKhRK11page_size_tmP16mem_block_info_t+0x197)[0x11a95b7]
/usr/sbin/mysqld[0x10ee0f2]
/usr/sbin/mysqld[0x10f167e]
/usr/sbin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x2ea7)[0x10f56e7]
/usr/sbin/mysqld(_ZN11ha_innobase13general_fetchEPhjj+0x1b8)[0xfdb088]
/usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0xfc)[0x7ff62c]
/usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x20)[0xc28820]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x147)[0xc9a917]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x267)[0xc934c7]
/usr/sbin/mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x17d)[0xd04cad]
/usr/sbin/mysqld[0x75a0d9]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x482e)[0xcc64de]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x5e5)[0xcc9805]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xabd)[0xcca3cd]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1df)[0xccbdef]
/usr/sbin/mysqld(handle_connection+0x2b8)[0xd94498]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf0e1a4]
/lib64/libpthread.so.0(+0x7e25)[0x7f261e1dde25]
/lib64/libc.so.6(clone+0x6d)[0x7f261c3aebad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f258806c050): select * from b_crm_act
Connection ID (thread ID): 60
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
2018-06-05T19:55:49.288858Z 0 [Warning] Changed limits: max_open_files: 5000 (requested 16246)
2018-06-05T19:55:49.289135Z 0 [Warning] Changed limits: table_open_cache: 2473 (requested 8096)
2018-06-05T19:55:49.473843Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2018-06-05T19:55:49.475922Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.21-21-log) starting as process 24691 ...
2018-06-05T19:55:49.484137Z 0 [Note] InnoDB: PUNCH HOLE support available
2018-06-05T19:55:49.484196Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-05T19:55:49.484203Z 0 [Note] InnoDB: Uses event mutexes
2018-06-05T19:55:49.484208Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-05T19:55:49.484217Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2018-06-05T19:55:49.484222Z 0 [Note] InnoDB: Using Linux native AIO
2018-06-05T19:55:49.484490Z 0 [Note] InnoDB: Number of pools: 1
2018-06-05T19:55:49.484623Z 0 [Note] InnoDB: Using CPU crc32 instructions
2018-06-05T19:55:49.487076Z 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2018-06-05T19:55:49.525048Z 0 [Note] InnoDB: Completed initialization of buffer pool
2018-06-05T19:55:49.536145Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-06-05T19:55:49.593791Z 0 [Note] InnoDB: Recovering partial pages from the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2018-06-05T19:55:50.007459Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-05T19:55:50.017727Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 18535505204
2018-06-05T19:55:50.017834Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 18535505213
2018-06-05T19:55:50.017878Z 0 [Note] InnoDB: Database was not shutdown normally!
2018-06-05T19:55:50.017912Z 0 [Note] InnoDB: Starting crash recovery.
2018-06-05T19:55:52.371846Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 31457280 bytes
2018-06-05T19:55:53.181733Z 0 [Note] InnoDB: Last MySQL binlog file position 0 81936, file name mysql-bin.000046
2018-06-05T19:55:53.463961Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-06-05T19:55:53.464156Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-06-05T19:55:53.464316Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2018-06-05T19:55:53.702925Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2018-06-05T19:55:53.706512Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2018-06-05T19:55:53.706579Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2018-06-05T19:55:53.707025Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4171ms. The settings might not be optimal. (flushed=0, during the time.)
2018-06-05T19:55:53.711042Z 0 [Note] InnoDB: Waiting for purge to start
2018-06-05T19:55:53.773707Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.21-21 started; log sequence number 18535505213
2018-06-05T19:55:53.776639Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2018-06-05T19:55:53.777106Z 0 [Note] Plugin 'FEDERATED' is disabled.
2018-06-05T19:55:53.919767Z 0 [Note] Recovering after a crash using mysql-bin
2018-06-05T19:55:53.921458Z 0 [Note] Starting crash recovery...
2018-06-05T19:55:53.921531Z 0 [Note] Crash recovery finished.
2018-06-05T19:55:54.161638Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2018-06-05T19:55:54.161717Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2018-06-05T19:55:54.164286Z 0 [Warning] CA certificate ca.pem is self signed.
2018-06-05T19:55:54.164552Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2018-06-05T19:55:54.164912Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2018-06-05T19:55:54.165163Z 0 [Note] IPv6 is available.
2018-06-05T19:55:54.165220Z 0 [Note]   - '::' resolves to '::';
2018-06-05T19:55:54.165271Z 0 [Note] Server socket created on IP: '::'.
2018-06-05T19:55:54.220166Z 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.220246Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.220274Z 0 [Warning] 'user' entry 'bitrix0@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.220311Z 0 [Warning] 'user' entry 'bx_clusteruser@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.220348Z 0 [Warning] 'user' entry 'bx_repluser@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.220415Z 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.220431Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.220442Z 0 [Warning] 'db' entry 'sitemanager bitrix0@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.220473Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.299674Z 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.299788Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-06-05T19:55:54.360989Z 0 [Note] Event Scheduler: Loaded 0 events
2018-06-05T19:55:54.362018Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.21-21-log'  socket: '/var/lib/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 21, Revision 2a37e4e
2018-06-05T19:55:56.330283Z 0 [Note] InnoDB: Buffer pool(s) load completed at 180605 22:55:56

3. Чек БД и таблиц показывает ОК

Comments

[neonox]

Конфиг не влез в вопрос

my.cnf

[client]
port = 3306
socket = /var/lib/mysqld/mysqld.sock
default-character-set = utf8
[mysqld_safe]
nice = 0
socket = /var/lib/mysqld/mysqld.sock
[mysqld]
# Basic mysql server configuration
user = mysql
port = 3306
basedir = /usr
datadir = /var/lib/mysql
socket = /var/lib/mysqld/mysqld.sock
skip-external-locking
default-storage-engine = innodb
pid-file = /var/run/mysqld/mysqld.pid
transaction-isolation = READ-COMMITTED
max_allowed_packet = 16M
myisam-recover-options = BACKUP
explicit_defaults_for_timestamp = 1
sql_mode = ""
expire_logs_days = 10
max_binlog_size = 100M
# Cache parameters
query_cache_size = 32M
table_open_cache = 4096
thread_cache_size = 32
key_buffer_size = 16M
thread_stack = 512K
join_buffer_size = 2M
sort_buffer_size = 2M
# Parameters for temporary tables
tmpdir = /tmp
max_heap_table_size	= 32M
tmp_table_size = 32M
# InnoDB parameters
innodb_file_per_table
innodb_buffer_pool_size	= 2048M
innodb_flush_log_at_trx_commit = 2
innodb_log_file_size = 512M
innodb_flush_method = O_DIRECT
# Database charset parameters
character-set-server = utf8
collation-server = utf8_unicode_ci
init-connect = "SET NAMES utf8 COLLATE utf8_unicode_ci"
skip-character-set-client-handshake
skip-name-resolve
[mysqldump]
quick
quote-names
max_allowed_packet = 1024M
[mysql]
[isamchk]
key_buffer = 16M
# Include additional settings
!includedir /etc/mysql/conf.d/

[entermix]

Pastebin? Не, не слышал?

Answer 1

[Reversaidx]

Попробуй увеличить max_allowed_packet
В этой дире includedir /etc/mysql/conf.d/ точно ничего нет?
Возможно где-то в этой инклуде прописаны таймауты, попробуй их тоже поднять

Comments

[neonox]

Не, не помогло(
Еще эксперементальным путем дошел до того, что select * from b_crm_act limit 3000; не вызывает падение БД.
А если выставить лимит в 4000, то уже происходит падение

Answer 2

[Руслан Федосеев]

mysqlcheck прогоните по серверу

Answer 3

[Андрей]

Не, не помогло(

ты в какой секции увеличивал max_allowed_packet ?
увеличь в секции
[mysqld] и в [mysqldump] ещё и таймауты надо увеличить будет скорее всего.

mysql> show variables like '%time%';

что выдаёт?