Как ускорить запросы Update Mysql 25 млн. строк?

Question

[tvoyadres]

Есть таблица где хранятся просмотры страниц

CREATE TABLE `hi_hits` (
  `id` int(10) UNSIGNED NOT NULL,
  `user` int(10) UNSIGNED NOT NULL DEFAULT 0,
  `url` varchar(255) DEFAULT NULL,
  `hits` mediumint(5) UNSIGNED NOT NULL DEFAULT 0,
  `calls` tinyint(2) UNSIGNED NOT NULL DEFAULT 0,
  `referer` varchar(255) DEFAULT NULL,
  `ip` int(10) UNSIGNED NOT NULL DEFAULT 0,
  `date_added` datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8
PARTITION BY KEY (`ip`)
PARTITIONS 100;

--
-- Индексы сохранённых таблиц
--

--
-- Индексы таблицы `hi_hits`
--
ALTER TABLE `hi_hits`
  ADD PRIMARY KEY (`id`,`ip`) USING BTREE,
  ADD KEY `url` (`url`),
  ADD KEY `date_added` (`date_added`) USING BTREE;

Периодически жутко тормозят Update, бывает до 20 секунд доходят, сама база MariaDB 10.4 находиться на SSD Samsung SM951

Может кто подскажет как выкрутиться?

Comments

[Melkij]

а какие update делаете надо догадаться? Особенно по части where.

[tvoyadres]

Melkij,

UPDATE hi_hits SET `hits` = "20", `date_added` = "2020-04-28 17:07:47" WHERE `id` = "424618376" AND `ip` = "3232235777" LIMIT 1

Тормозит не постоянно, а как-то странно периодически аналогичные запросы выполнятся обычно за 0.001 сек, но бывает и от 1 до 20 сек

[tvoyadres]

Может SSD тормозит?

Если на Centos 7 запустить команду

iostat -xm -t 5

то колонка %util доходит бывает до 15

[Дмитрий Шицков]

Возможно, часть индексов не нужна и лишь замедляет UPDATE.

А дивжок какой использован?
Селекты и апдейты с какими explain?

[Максим]

tvoyadres, Может кто то еще делает update (lock) и у Вас возникают lock wait?

[tvoyadres]

Максим, а как узнать это?

[tvoyadres]

Дмитрий Шицков,

EXPLAIN UPDATE hi_hits SET `hits` = "20", `date_added` = "2020-04-28 17:07:47" WHERE `id` = "424618376" AND `ip` = "3232235777" LIMIT 1

1 SIMPLE hi_hits range PRIMARY PRIMARY 8 NULL 1 Using where

EXPLAIN SELECT * FROM `hi_hits` WHERE `id` = "424618376" AND `ip` = "3232235777" LIMIT 1

1 SIMPLE hi_hits const PRIMARY PRIMARY 8 const,const 1

[Дмитрий Шицков]

tvoyadres, я имел ввиду EXPLAIN ANALYZE, в этих explain про план запроса вообще ничего)

[tvoyadres]

Дмитрий Шицков, что-то не получается выполнить запрос в phpmyadmin

[Everything_is_not_so_bad]

tvoyadres, индексы вроде бы верные.

что-то не получается выполнить запрос в phpmyadmin

Попробуйте в клиенте mysql с пользователем root или другими административными правами.

[tvoyadres]

Роман Мирр,
Зашел в командную строку получаю ошибку

MariaDB [tvoyadres]> EXPLAIN ANALYZE SELECT * FROM `hi_hits` WHERE `id` = "424618376" AND `ip` = "3232235777" LIMIT 1\G
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'ANALYZE SELECT * FROM `hi_hits` WHERE `id` = "424618376" AND `ip` = "3232235777"' at line 1

[Everything_is_not_so_bad]

tvoyadres, слово EXPLAIN лишнее у MariaDB.

[tvoyadres]

Роман Мирр, О спасибо

ANALYZE SELECT * FROM `hi_hits` WHERE `id` = "424618376" AND `ip` = "3232235777" LIMIT 1\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: hi_hits
         type: const
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 8
          ref: const,const
         rows: 1
       r_rows: NULL
     filtered: 100.00
   r_filtered: NULL
        Extra: 
1 row in set (0.002 sec)

ANALYZE UPDATE `hi_hits` SET `hits` = "24", `date_added` = "2020-04-28 17:07:47" WHERE `id` = "424618376" AND `ip` = "3232235777" LIMIT 1\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: hi_hits
         type: range
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 8
          ref: NULL
         rows: 1
       r_rows: 1.00
     filtered: 100.00
   r_filtered: 100.00
        Extra: Using where
1 row in set (0.000 sec)

Answer 1

[Леонид]

убедитесь что есть индексы на полях которые в условии where при апдейте

Answer 2

[Everything_is_not_so_bad]

Можно периодически проверять наличие открытых таблиц:
show open tables where In_Use > 0
Если непустой вывод, тогда эти таблицы используются в тот момент. Пока таблица заблокирована для обновления, другие запросы ждут.
Добавлено
Если используются транзакции, то они запирают таблицу на время обновления.
Также возможно, что ресурсов недостаточно на машине.
Проверить рекомендации mysqltuner.

Comments

[tvoyadres]

Часто бывает не пустой и доходит до 7

[tvoyadres]

mysqltuner
>> MySQLTuner 1.7.13 - Major Hayden
>> Bug reports, feature requests, and downloads at mysqltuner.com
>> Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
Please enter your MySQL administrative login: root
Please enter your MySQL administrative password: [OK] Currently running supported MySQL version 10.4.12-MariaDB
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/log/mariadb/mariadb.log(1M)
[OK] Log file /var/log/mariadb/mariadb.log exists
[OK] Log file /var/log/mariadb/mariadb.log is readable.
[OK] Log file /var/log/mariadb/mariadb.log is not empty
[OK] Log file /var/log/mariadb/mariadb.log is smaller than 32 Mb
[!!] /var/log/mariadb/mariadb.log contains 15746 warning(s).
[!!] /var/log/mariadb/mariadb.log contains 7 error(s).
[--] 11 start(s) detected in /var/log/mariadb/mariadb.log
[--] 1) 2020-04-28 13:55:00 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 2) 2020-04-10 22:11:05 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 3) 2020-03-31 21:39:15 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 4) 2020-03-28 18:29:50 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 5) 2020-03-11 21:13:30 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 6) 2020-03-09 17:11:20 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 7) 2020-03-09 16:07:02 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 8) 2020-03-09 15:54:36 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 9) 2020-03-05 20:41:57 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 10) 2020-03-01 12:28:49 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 9 shutdown(s) detected in /var/log/mariadb/mariadb.log
[--] 1) 2020-04-28 13:54:59 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 2) 2020-04-10 22:08:49 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 3) 2020-03-31 21:38:38 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 4) 2020-03-28 18:29:49 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 5) 2020-03-11 21:09:16 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 6) 2020-03-09 17:10:31 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 7) 2020-03-09 15:56:28 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 8) 2020-03-05 20:37:52 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 9) 2020-03-01 12:28:48 0 [Note] /usr/sbin/mysqld: Shutdown complete

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in InnoDB tables: 16.6G (Tables: 268)
[!!] Total fragmented tables: 2

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!

-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 23h 6m 11s (50M q [603.921 qps], 1M conn, TX: 69G, RX: 17G)
[--] Reads / Writes: 94% / 6%
[--] Binary logging is disabled
[--] Physical Memory : 62.8G
[--] Max MySQL memory : 36.3G
[--] Other process memory: 2.1G
[--] Total buffers: 33.3G global + 8.0M per thread (384 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 35.3G (56.29% of installed RAM)
[OK] Maximum possible memory usage: 36.3G (57.88% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (95K/50M)
[OK] Highest usage of available connections: 66% (256/384)
[OK] Aborted connections: 0.01% (95/1254761)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 10.4% (5M cached / 48M selects)
[!!] Query cache prunes per day: 17157405
[OK] Sorts requiring temporary tables: 0% (5K temp sorts / 2M sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 1% (1K on disk / 132K total)
[OK] Thread cache hit rate: 99% (256 created / 1M connections)
[OK] Table cache hit rate: 97% (672 open / 687 opened)
[OK] Open file limit used: 0% (30/32K)
[OK] Table locks acquired immediately: 100% (697 immediate / 697 locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 32 thread(s).
[--] Using default value is good enough for your version (10.4.12-MariaDB)

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (24M used / 134M cache)
[!!] Cannot calculate MyISAM index size - re-run script as root user

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 32.0G/16.6G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (6.25 %): 1.0G * 2/32.0G should be equal 25%
[OK] InnoDB buffer pool instances: 32
[--] Number of InnoDB Buffer Pool Chunk : 256 for 32 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (73657064441 hits/ 73657909852 total)
[!!] InnoDB Write Log efficiency: 77.58% (6655076 hits/ 8578729 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1923653 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/304.0K
[OK] Aria pagecache hit rate: 97.4% (72K cached / 1K reads)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
Control warning line(s) into /var/log/mariadb/mariadb.log file
Control error line(s) into /var/log/mariadb/mariadb.log file
Run OPTIMIZE TABLE to defragment tables for better performance
OPTIMIZE TABLE `tvoyadres`.`hi_files`; -- can free 448 MB
OPTIMIZE TABLE `tvoyadres`.`hi_materials`; -- can free 478 MB
Total freed space after theses OPTIMIZE TABLE : 926 Mb
MySQL was started within the last 24 hours - recommendations may be inaccurate
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
Performance schema should be activated for better diagnostics
Consider installing Sys schema from https://github.com/mysql/mysql-sys
Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
query_cache_size (=0)
query_cache_type (=0)
query_cache_limit (> 4M, or use smaller result sets)
query_cache_size (> 64M)
performance_schema = ON enable PFS
innodb_log_file_size should be (=4G) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

[Everything_is_not_so_bad]

tvoyadres, нужно обратить внимание на строки с [!!]
плюс рекомендации

-------- Recommendations ---------------------------------------------------------------------------
...