У меня питон вылетает после 600 вставок записи в кликхаус.
tracemalloc.start()
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
snapshot1 = tracemalloc.take_snapshot()
for row in rows:
values = list(row.values()) # 94
keys = list(row.keys()) # 95
ret = client.insert(table, [values], column_names=keys) # 96
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
for stat in top_stats[:10]:
print(stat)
tracemalloc после 100 проходов цикла распечатал следующее.
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/context.py:24: size=25.8 KiB +25.8 KiB), count=171 (+171), average=155 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/client.py:651: size=15.9 KiB +15.9 KiB), count=115 (+115), average=142 B
/root/sync/dbsync.py:95: size=14.2 KiB (+14.2 KiB), count=114 (+114), average=127 B
/root/sync/dbsync.py:94: size=14.2 KiB (+14.2 KiB), count=114 (+114), average=127 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/query.py:293: size=4416 B (+416 B), count=46 (+46), average=96 B
/root/sync/dbsync.py:96: size=4176 B (+4176 B), count=115 (+115), average=36 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/insert.py:87: size=3648 B (+348 B), count=57 (+57), average=64 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/context.py:42: size=3648 B (+648 B), count=57 (+57), average=64 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/context.py:41: size=3648 B (+648 B), count=57 (+57), average=64 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/client.py:656: size=3248 B (+248 B), count=58 (+58), average=56 B
Я проставил коментами номера строк файла dbsync.py
Отдельный вопрос, который, вероятно надо задавать не здесь - почему утечки в модуле clickhouse_connect, но почему также утечки в моем коде строки 94 95 96?
Сообщение об ошибке
Error during insert: :HTTPDriver for http://localhost:8123 returned response code 500)
Code: 241. DB::Exception: Memory limit (total) exceeded: would use 1.01 GiB (attempt to allocate chunk of 5039783 bytes), maximum: 861.77 MiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker. (MEMORY_LIMIT_EXCEEDED) (version 23.11.2.11 (official build))
Traceback (most recent call last):
File "/root/sync/dbsync.py", line 97, in <module>
ret = client.insert(table, [values], column_names=keys)
File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/client.py", line 554, in insert
return self.data_insert(context)
File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/httpclient.py", line 250, in data_insert
response = self._raw_request(block_gen, params, headers, error_handler=error_handler, server_wait=False)
File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/httpclient.py", line 431, in _raw_request
error_handler(response)
File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/httpclient.py", line 236, in error_handler
self._error_handler(resp)
File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/httpclient.py", line 357, in _error_handler
raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
clickhouse_connect.driver.exceptions.DatabaseError: :HTTPDriver for http://localhost:8123 returned response code 500)
Code: 241. DB::Exception: Memory limit (total) exceeded: would use 1.01 GiB (attempt to allocate chunk of 5039783 bytes), maximum: 861.77 MiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker. (MEMORY_LIMIT_EXCEEDED) (version 23.11.2.11 (official build))
