Vitaly Karasik, Понял спасибо, попрошу обновить. График по памяти был стабильный, если достаточно того что в ISP рисует. По перегреву там и раньше какая то проблема была, устраняли они.
Настройки по умолчанию в основном. Сейчас убрал определение rt индексов и запараллелил свою тест среду, что бы в mysql и sphinx одновременно искали( все таки сервер под на нагрузкой).
В итоге получаю почти стабильно гораздо лучше показатели у sphinx
Пума Тайланд, по isp все ровно, расход более 50% по оперативке не показывает, может конечно это средние значения. и память в какой-то момент жестко утекает. все время эти 15 мин еще, странно..
[Thu Nov 1 06:54:01.880 2018] [20587] FATAL: out of memory (unable to allocate 56026638920 bytes)
[Thu Nov 1 06:54:01.930 2018] [24725] watchdog: main process 20578 crashed via CRASH_EXIT (exit code 2), will be restarted
[Thu Nov 1 06:54:01.931 2018] [24725] watchdog: main process 20588 forked ok
[Thu Nov 1 06:54:01.935 2018] [20588] listening on all interfaces, port=9312
[Thu Nov 1 06:54:02.117 2018] [20588] accepting connections
[Thu Nov 1 06:54:02.385 2018] [20613] FATAL: out of memory (unable to allocate 56026638920 bytes)
[Thu Nov 1 06:54:02.417 2018] [24725] watchdog: main process 20588 crashed via CRASH_EXIT (exit code 2), will be restarted
[Thu Nov 1 06:54:02.417 2018] [24725] watchdog: main process 20614 forked ok
[Thu Nov 1 06:54:02.422 2018] [20614] listening on all interfaces, port=9312
[Thu Nov 1 06:54:02.602 2018] [20614] accepting connections
[Thu Nov 1 06:54:04.350 2018] [20676] FATAL: out of memory (unable to allocate 56026638920 bytes)
------- FATAL: CRASH DUMP -------
[Thu Nov 1 06:54:04.348 2018] [20614]
--- crashed SphinxAPI request dump ---
--- request dump end ---
Sphinx 2.2.11-id64-release (95ae9a6)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with gcc 4.8.2
Configured with flags: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc/sphinx' '--with-mysql' '--with-re2' '--with-libstemmer' '--with-unixodbc' '--with-iconv' '--enable-id64' '--with-pgsql' '--with-syslog' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'
Host OS is Linux localhost.localdomain 3.10.0-123.6.3.el7.x86_64 #1 SMP Wed Jul 16 15:10:46 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7ffe846cc9b7, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7ffe846cad30)
Stack looks OK, attempting backtrace.
0x4106d2
0x42023a
0x45af70
0x45c0f6
0x4091f6
0x7fcc2d907c05
Something wrong in frame pointers, manual backtrace failed (fp=0)
Trying system backtrace:
[Thu Nov 1 06:54:04.387 2018] [24725] watchdog: main process 20614 crashed via CRASH_EXIT (exit code 2), will be restarted
[Thu Nov 1 06:54:04.387 2018] [24725] watchdog: main process 20677 forked ok
[Thu Nov 1 06:54:04.391 2018] [20677] listening on all interfaces, port=9312
[Thu Nov 1 06:54:04.570 2018] [20677] accepting connections
[Thu Nov 1 06:54:05.719 2018] [20708] FATAL: out of memory (unable to allocate 56026638920 bytes)
[Thu Nov 1 06:54:05.751 2018] [24725] watchdog: main process 20677 crashed via CRASH_EXIT (exit code 2), will be restarted
[Thu Nov 1 06:54:05.751 2018] [24725] watchdog: main process 20709 forked ok
[Thu Nov 1 06:54:05.755 2018] [20709] listening on all interfaces, port=9312
[Thu Nov 1 06:54:05.941 2018] [20709] accepting connections
[Thu Nov 1 06:54:06.553 2018] [20736] FATAL: out of memory (unable to allocate 56026638920 bytes)
[Thu Nov 1 06:54:06.591 2018] [24725] watchdog: main process 20709 crashed via CRASH_EXIT (exit code 2), will be restarted
[Thu Nov 1 06:54:06.591 2018] [24725] watchdog: main process 20737 forked ok
[Thu Nov 1 06:54:06.595 2018] [20737] listening on all interfaces, port=9312
[Thu Nov 1 06:54:06.775 2018] [20737] accepting connections
[Thu Nov 1 06:54:06.998 2018] [20753] FATAL: out of memory (unable to allocate 56026638920 bytes)
------- FATAL: CRASH DUMP -------
[Thu Nov 1 06:54:06.997 2018] [20737]
--- crashed SphinxAPI request dump ---
--- request dump end ---
Sphinx 2.2.11-id64-release (95ae9a6)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with gcc 4.8.2
Configured with flags: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc/sphinx' '--with-mysql' '--with-re2' '--with-libstemmer' '--with-unixodbc' '--with-iconv' '--enable-id64' '--with-pgsql' '--with-syslog' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'
Host OS is Linux localhost.localdomain 3.10.0-123.6.3.el7.x86_64 #1 SMP Wed Jul 16 15:10:46 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7fcc1b1f3df7, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7fcc1b1f25f0)
Stack looks OK, attempting backtrace.
0x4106d2
0x41ce29
0x45a6b2
0x45ae4c
0x41046e
0x57970c
0x7fcc2f886e25
Something wrong in frame pointers, manual backtrace failed (fp=0)
Trying system backtrace:
[Thu Nov 1 06:54:07.035 2018] [24725] watchdog: main process 20737 crashed via CRASH_EXIT (exit code 2), will be restarted
[Thu Nov 1 06:54:07.036 2018] [24725] watchdog: main process 20755 forked ok
[Thu Nov 1 06:54:07.040 2018] [20755] listening on all interfaces, port=9312
[Thu Nov 1 06:54:07.251 2018] [20755] accepting connections
[Thu Nov 1 06:54:07.549 2018] [20771] FATAL: out of memory (unable to allocate 56026638920 bytes)
[Thu Nov 1 06:54:07.583 2018] [24725] watchdog: main process 20755 crashed via CRASH_EXIT (exit code 2), will be restarted
[Thu Nov 1 06:54:15.330 2018] [24725] watchdog: main process 20990 forked ok
[Thu Nov 1 06:54:15.334 2018] [20990] FATAL: failed to create TLS for connection ID
[Thu Nov 1 06:54:15.335 2018] [24725] watchdog: main process 20990 exited cleanly (exit code 1), shutting down
[Thu Nov 1 07:00:01.532 2018] [23759] watchdog: main process 23760 forked ok
[Thu Nov 1 07:00:01.534 2018] [23760] listening on all interfaces, port=9312
[Thu Nov 1 07:00:02.698 2018] [23760] accepting connections
[Thu Nov 1 07:15:01.822 2018] [23759] watchdog: main process 23760 killed cleanly with SIGKILL, shutting down
[Thu Nov 1 07:15:01.839 2018] [7757] watchdog: main process 7758 forked ok
[Thu Nov 1 07:15:01.840 2018] [7758] listening on all interfaces, port=9312
[Thu Nov 1 07:15:02.887 2018] [7758] accepting connections
[Thu Nov 1 07:30:01.958 2018] [7757] watchdog: main process 7758 killed cleanly with SIGKILL, shutting down
[Thu Nov 1 07:30:01.993 2018] [24039] watchdog: main process 24040 forked ok
[Thu Nov 1 07:30:01.994 2018] [24040] listening on all interfaces, port=9312
[Thu Nov 1 07:30:03.128 2018] [24040] accepting connections
[Thu Nov 1 07:45:01.737 2018] [24039] watchdog: main process 24040 killed cleanly with SIGKILL, shutting down
[Thu Nov 1 07:45:01.791 2018] [6713] watchdog: main process 6714 forked ok
[Thu Nov 1 07:45:01.791 2018] [6714] listening on all interfaces, port=9312
[Thu Nov 1 07:45:03.208 2018] [6714] accepting connections
[Thu Nov 1 08:00:01.985 2018] [6713] watchdog: main process 6714 killed cleanly with SIGKILL, shutting down
[Thu Nov 1 08:00:01.995 2018] [21991] watchdog: main process 21992 forked ok
[Thu Nov 1 08:00:01.997 2018] [21992] listening on all interfaces, port=9312
[Thu Nov 1 08:00:03.321 2018] [21992] accepting connections
[Thu Nov 1 08:15:01.539 2018] [21991] watchdog: main process 21992 killed cleanly with SIGKILL, shutting down
[Thu Nov 1 08:15:01.555 2018] [3928] watchdog: main process 3929 forked ok
[Thu Nov 1 08:15:01.556 2018] [3929] listening on all interfaces, port=9312
[Thu Nov 1 08:15:02.647 2018] [3929] accepting connections
[Thu Nov 1 08:30:01.832 2018] [3928] watchdog: main process 3929 killed cleanly with SIGKILL, shutting down
[Thu Nov 1 08:30:01.875 2018] [19531] watchdog: main process 19532 forked ok
[Thu Nov 1 08:30:01.876 2018] [19532] listening on all interfaces, port=9312
[Thu Nov 1 08:30:03.567 2018] [19532] accepting connections
Кирилл Несмеянов, это интуитивно мне понятно, так и есть наверно, searchd висит на мониторинге в isp, трафик большой может и дублируется поэтому. Вопрос почему он перезапускается каждые 15 мин. За день работы лог пополнился новым типом ошибок, прикрепляю на всякий случай.
mysqltuner говорит innodb_buffer_pool_size (>= 64.3G) if possible.