Как решить проблему с нехваткой памяти у сервера?

Question

[Hdeul]

Как решить проблему с нехваткой памяти?

У меня есть сервер 8 GB Memory / 25 GB Disk

На нем запущенны 3 бота телеграм, 2 воркера и одна на минутку.
У меня постоянно база данных падает из нехватка памяти, у меня уже было похожее ситуации когда трафик на сервере упал, пришлось докупать сервер с 5 долларов до 50 долларов.

Но сейчас ситуации совсем другая, на этот раз нехватка памяти.

Я думаю может как то докер неправильно настроил или нужно покупать сервер с больше памяти?
Или может отказаться от докеров?

Comments

[Anonymous]

У вас настроен swap?

[Дмитрий]

зачем тут докер не ясно

Answer 1

[ky0]

Гарантированный объём памяти, не зависящий от других процессов, можно получить, перенеся базу из контейнера в виртуальную машину с честным, без балунинга, выделением памяти.

С другой стороны - постгрес, например, потребляет тот объём, который задан ему в настройках и сверх того памяти не требует.

Comments

[Hdeul]

У меня база и так не зависит от контейнера

[ky0]

Hdeul, если база работает по соседству с другими процессами, то если последние отожрут много памяти - база совершенно справедливо упадёт. Виртуалка, если её соответствующим образом настроить, себе забирает всю память сразу и никакие другие процессы ей затем не страшны.

[Hdeul]

ky0, хорошо, я как понял нужно тут свап настроить или внутри докера можно настроить?

Answer 2

[mayton2019]

Очень странно что падает база. Обычно чаще приложения падают. Мне кажется что базу можно завалить бесконтрольным ростом числа сессий (коннектов) или долгими сортировками одной большой таблицы. Здесь надо смотреть детальнее. Что там за база? Что пишет в логах перед падением?

Вобщем я-бы пошел не от докеров и от железа а от проблемы. Проблема - то как используются ресурсы базы.

Comments

[Hdeul]

```2022-07-29 13:37:13.328 UTC [526422] cosmos_ton@cosmos_ton LOG: could not receive data from client: Connection reset by peer
2022-07-29 13:37:13.339 UTC [526432] cosmos_ton@cosmos_ton LOG: could not receive data from client: Connection reset by peer
2022-07-29 13:37:13.356 UTC [527297] cosmos_ton@cosmos_ton LOG: could not receive data from client: Connection reset by peer
2022-07-29 13:37:13.364 UTC [526373] cosmos_ton@cosmos_ton LOG: could not receive data from client: Connection reset by peer
2022-07-29 17:22:34.474 UTC [532092] [unknown]@[unknown] LOG: could not accept SSL connection: unsupported protocol
2022-07-29 17:22:34.474 UTC [532092] [unknown]@[unknown] HINT: This may indicate that the client does not support any SSL protocol version between TLSv1.2 and TLSv1.3.
2022-07-30 00:24:27.206 UTC [544722] postgres@bbbbbbb FATAL: no pg_hba.conf entry for host "185.7.214.104", user "postgres", database "bbbbbbb", no encryption
2022-07-30 05:22:43.550 UTC [531562] cosmos_ton@cosmos_ton LOG: could not receive data from client: Connection reset by peer
2022-07-30 06:33:55.245 UTC [559281] postgres@postgres FATAL: no pg_hba.conf entry for host "34.77.127.183", user "postgres", database "postgres", SSL encryption
2022-07-30 06:33:55.267 UTC [559282] postgres@postgres FATAL: no pg_hba.conf entry for host "34.77.127.183", user "postgres", database "postgres", no encryption
2022-07-30 06:47:33.422 UTC [559552] [unknown]@[unknown] LOG: invalid length of startup packet
2022-07-30 08:03:52.649 UTC [560786] [unknown]@[unknown] FATAL: unsupported frontend protocol 16.0: server supports 3.0 to 3.0
2022-07-30 08:10:00.587 UTC [561022] [unknown]@[unknown] LOG: invalid length of startup packet
2022-07-30 09:26:38.657 UTC [438380] PANIC: could not write to file "pg_logical/replorigin_checkpoint.tmp": No space left on device
2022-07-30 09:26:39.120 UTC [438378] LOG: checkpointer process (PID 438380) was terminated by signal 6: Aborted
2022-07-30 09:26:39.121 UTC [438378] LOG: terminating any other active server processes
2022-07-30 09:26:39.142 UTC [438384] LOG: could not write temporary statistics file "pg_stat/db_17120.tmp": No space left on device
2022-07-30 09:26:39.143 UTC [438384] LOG: could not write temporary statistics file "pg_stat/db_16386.tmp": No space left on device
2022-07-30 09:26:39.143 UTC [438384] LOG: could not close temporary statistics file "pg_stat/db_1.tmp": No space left on device
2022-07-30 09:26:39.143 UTC [438384] LOG: could not close temporary statistics file "pg_stat/db_0.2022-07-30 09:26:39.744 UTC [563733] LOG: database system was not properly shut down; automatic recovery in progress
2022-07-30 09:26:39.749 UTC [563733] LOG: redo starts at 0/2630E750
2022-07-30 09:26:39.750 UTC [563733] LOG: invalid record length at 0/26313080: wanted 24, got 0
2022-07-30 09:26:39.751 UTC [563733] LOG: redo done at 0/26313048 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2022-07-30 09:26:39.780 UTC [438378] LOG: database system is ready to accept connections
2022-07-30 09:26:41.825 UTC [563757] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:26:47.624 UTC [563758] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:26:51.858 UTC [563759] cosmos_ton@cosmos_ton WARNING: could not create relation-cache initialization file "global/pg_internal.init.563759": No space left on device
2022-07-30 09:26:51.858 UTC [563759] cosmos_ton@cosmos_ton DETAIL: Continuing anyway, but there's something wrong.
2022-07-30 09:26:51.858 UTC [563759] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:26:57.631 UTC [563760] cosmos_ton@cosmos_ton WARNING: could not create relation-cache initialization file "global/pg_internal.init.563760": No space left on device
2022-07-30 09:26:57.631 UTC [563760] cosmos_ton@cosmos_ton DETAIL: Continuing anyway, but there's something wrong.
2022-07-30 09:26:57.631 UTC [563760] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:01.854 UTC [563761] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:03.081 UTC [563762] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:07.643 UTC [563765] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:11.838 UTC [563767] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:17.639 UTC [563768] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:21.862 UTC [563769] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:27.659 UTC [563772] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:31.866 UTC [563777] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:37.667 UTC [563780] cosmos_ton@cosmos_ton WARNING: could not create relation-cache initialization file "global/pg_internal.init.563780": No space left on device
2022-07-30 09:27:37.667 UTC [563780] cosmos_ton@cosmos_ton DETAIL: Continuing anyway, but there's something wrong.
2022-07-30 09:27:37.669 UTC [563780] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:39.830 UTC [563782] FATAL: could not write init file
2022-07-30 09:27:41.900 UTC [563784] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:47.687 UTC [563785] cosmos_ton@cosmos_ton WARNING: could not create relation-cache initialization file "global/pg_internal.init.563785": No space left on device
2022-07-30 09:27:47.687 UTC [563785] cosmos_ton@cosmos_ton DETAIL: Continuing anyway, but there's something wrong.
2022-07-30 09:27:47.687 UTC [563785] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:51.881 UTC [563786] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:27:54.127 UTC [563787] toncoin@toncoin_bot FATAL: could not write init file
2022-07-30 09:27:57.653 UTC [563788] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:28:01.928 UTC [563791] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:28:03.089 UTC [563792] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:28:07.669 UTC [563794] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:28:11.896 UTC [563795] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:28:17.679 UTC [563796] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:28:21.914 UTC [563799] cosmos_ton@cosmos_ton WARNING: could not create relation-cache initialization file "global/pg_internal.init.563799": No space left on device
2022-07-30 09:28:21.9142022-07-30 13:01:44.207 UTC [438378] LOG: database system is shut down
``` вот логи ошибки бд

[Hdeul]

Но у меня параметры были такие Usage of /: 99% of 24.05GB IPv4 address for docker0: 172.17.0.1

[mayton2019]

Судя по этому

2022-07-30 09:28:17.679 UTC [563796] cosmos_ton@cosmos_ton FATAL: could not write init file
2022-07-30 09:28:21.914 UTC [563799] cosmos_ton@cosmos_ton WARNING: could not create relation-cache initialization file "global/pg_internal.init.563799": No space left on device
2022-07-30 09:28:21.9142022-07-30 13:01:44.207 UTC [438378] LOG: database system is shut down

У тебя не хватает дисковой памяти.

Если запускаешь постгресы в докерах - то тебе надо монтировать фолдер с датой как внешниее файловое хранилище. Тоесть база не должна лежать в докеровской OVERLAY системе. Она - медленная и не переживает ребут.

А это что? Вообще я тут не пойму но похоже на проблему security. Кто-то к тебе ходит кто не должен ходить.

2022-07-30 06:33:55.267 UTC [559282] postgres@postgres FATAL: no pg_hba.conf entry for host "34.77.127.183", user "postgres", database "postgres", no encryption
2022-07-30 06:47:33.422 UTC [559552] [unknown]@[unknown] LOG: invalid length of startup packet
2022-07-30 08:03:52.649 UTC [560786] [unknown]@[unknown] FATAL: unsupported frontend protocol 16.0: server supports 3.0 to 3.0