|
Hi experts, I'm using Patroni Postgres installation and noticed that twice already postgres crashed due to out of memory. I'm using logical replication with around 30-40 active subscribers on
this machine. The machine has 128GB but only 32GB is allocated to Postgres. How can I know what is actually causing the out of memory issue? Is it caused by not optimal postgres configuration or something else? /usr/lib/postgresql/13/bin/postgres -D /home/postgres/pgdata/pgroot/data --config-file=/home/postgres/pgdata/pgroot/data/postgresql.conf
--port=5432 --cluster_name=postgres-cluster --wal_level=logical --hot_standby=on --max_connections=533 --max_wal_senders=90 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=on --max_replication_slots=90 --max_worker_processes=30
--wal_log_hints=on Oct 27 07:05:31 node2 kernel: postgres invoked oom-killer: gfp_mask=0xd0, order=0,
oom_score_adj=993 Oct 27 07:05:31 node2 kernel: postgres cpuset=docker-6ae67e04710619972d3b1ab5d4c69c318d001c2da47fecee121cdc60279a14a0.scope mems_allowed=0 Oct 27 07:05:31 node2 kernel: CPU: 6 PID: 15536 Comm: postgres Kdump: loaded Tainted: G ------------ T 3.10.0-1160.el7.x86_64 #1 Oct 27 07:05:31 node2 kernel: Hardware name: Kontron MSP8040/4008, BIOS Core: 5.11, MSP804x: 1.57.0943FC77 05/06/2020 Oct 27 07:05:31 node2 kernel: Call Trace: Oct 27 07:05:31 node2 kernel: [<ffffffffa4581340>] dump_stack+0x19/0x1b Oct 27 07:05:31 node2 kernel: [<ffffffffa457bc60>] dump_header+0x90/0x229 Oct 27 07:05:31 node2 kernel: [<ffffffffa409d008>] ? ep_poll_callback+0xf8/0x220 Oct 27 07:05:31 node2 kernel: [<ffffffffa3fc1b26>] ? find_lock_task_mm+0x56/0xc0 Oct 27 07:05:31 node2 kernel: [<ffffffffa403c8c8>] ? try_get_mem_cgroup_from_mm+0x28/0x60 Oct 27 07:05:31 node2 kernel: [<ffffffffa3fc208d>] oom_kill_process+0x2cd/0x490 Oct 27 07:05:31 node2 kernel: [<ffffffffa4040cdc>] mem_cgroup_oom_synchronize+0x55c/0x590 Oct 27 07:05:31 node2 kernel: [<ffffffffa4040140>] ? mem_cgroup_charge_common+0xc0/0xc0 Oct 27 07:05:31 node2 kernel: [<ffffffffa3fc2974>] pagefault_out_of_memory+0x14/0x90 Oct 27 07:05:31 node2 kernel: [<ffffffffa457a16c>] mm_fault_error+0x6a/0x157 Oct 27 07:05:31 node2 kernel: [<ffffffffa458e8d1>] __do_page_fault+0x491/0x500 Oct 27 07:05:31 node2 kernel: [<ffffffffa458e975>] do_page_fault+0x35/0x90 Oct 27 07:05:31 node2 kernel: [<ffffffffa458a778>] page_fault+0x28/0x30 Oct 27 07:05:31 node2 kernel: Task in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5984f099_30b0_4506_8730_6c72e7c02b78.slice/docker-6ae67e0471061997$ Oct 27 07:05:31 node2 kernel: memory: usage 32768000kB, limit 32768000kB, failcnt 144867 Oct 27 07:05:31 node2 kernel: memory+swap: usage 32768000kB, limit 9007199254740988kB, failcnt 0 Oct 27 07:05:31 node2 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0 Oct 27 07:05:31 node2 kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5984f099_30b0_4506_8730_6c72e7c02b78.slice: cache:$ Oct 27 07:05:31 node2 kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5984f099_30b0_4506_8730_6c72e7c02b78.slice/docker-$ Oct 27 07:05:31 node2 kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5984f099_30b0_4506_8730_6c72e7c02b78.slice/docker-$ Oct 27 07:05:31 node2 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Oct 27 07:05:31 node2 kernel: [13159] 0 13159 239 1 3 0 -998 pause Oct 27 07:05:31 node2 kernel: [13322] 0 13322 1095 97 8 0 993 dumb-init Oct 27 07:05:31 node2 kernel: [13335] 0 13335 1156 171 8 0 993 sh Oct 27 07:05:31 node2 kernel: [13411] 0 13411 1137 98 8 0 993 runsvdir Oct 27 07:05:31 node2 kernel: [13438] 0 13438 1099 98 7 0 993 runsv Oct 27 07:05:31 node2 kernel: [13439] 0 13439 1099 98 7 0 993 runsv Oct 27 07:05:31 node2 kernel: [13440] 101 13440 27026 1186 54 0 993 pgqd Oct 27 07:05:31 node2 kernel: [13441] 101 13441 155215 8237 101 0 993 patroni Oct 27 07:05:31 node2 kernel: [19532] 101 19532 1740033 46817 171 0 993 postgres Oct 27 07:05:31 node2 kernel: [19542] 101 19542 1767874 6713 121 0 993 postgres Oct 27 07:05:31 node2 kernel: [19546] 101 19546 1740173 1445031 3166 0 993 postgres Oct 27 07:05:31 node2 kernel: [19547] 101 19547 1740069 20060 171 0 993 postgres Oct 27 07:05:31 node2 kernel: [19548] 101 19548 1740027 4821 86 0 993 postgres Oct 27 07:05:31 node2 kernel: [19549] 101 19549 1740283 1011 91 0 993 postgres Oct 27 07:05:31 node2 kernel: [19549] 101 19549 1740283 1011 91 0 993 postgres Oct 27 07:05:31 node2 kernel: [19550] 101 19550 50320 798 77 0 993 postgres Oct 27 07:05:31 node2 kernel: [19551] 101 19551 1740524 2097 117 0 993 postgres Oct 27 07:05:31 node2 kernel: [19552] 101 19552 1740766 2043 100 0 993 postgres Oct 27 07:05:31 node2 kernel: [19556] 101 19556 1744493 16306 476 0 993 postgres Oct 27 07:05:31 node2 kernel: [19588] 101 19588 1744522 16353 484 0 993 postgres Oct 27 07:05:31 node2 kernel: [19598] 101 19598 1745580 22539 602 0 993 postgres Oct 27 07:05:31 node2 kernel: [19609] 101 19609 5909368 4364063 9947 0 993 postgres Oct 27 07:05:31 node2 kernel: [19617] 101 19617 1745576 16669 494 0 993 postgres Oct 27 07:05:31 node2 kernel: [19676] 101 19676 1740468 2208 113 0 993 postgres Oct 27 07:05:31 node2 kernel: [19744] 101 19744 1741155 19698 615 0 993 postgres Oct 27 07:05:31 node2 kernel: [19776] 101 19776 1742008 105333 1733 0 993 postgres Oct 27 07:05:31 node2 kernel: [19793] 101 19793 1740738 58109 631 0 993 postgres Oct 27 07:05:31 node2 kernel: [19799] 101 19799 1740737 29227 546 0 993 postgres Oct 27 07:05:31 node2 kernel: [19806] 101 19806 1740435 1393 96 0 993 postgres Oct 27 07:05:31 node2 kernel: [19811] 101 19811 1741021 3658 139 0 993 postgres Oct 27 07:05:31 node2 kernel: [19971] 101 19971 1740734 15521 449 0 993 postgres Oct 27 07:05:31 node2 kernel: [20007] 101 20007 1740435 1394 96 0 993 postgres Oct 27 07:05:31 node2 kernel: [20537] 101 20537 1891639 121701 407 0 993 postgres Oct 27 07:05:31 node2 kernel: [20623] 101 20623 1889590 121300 405 0 993 postgres Oct 27 07:05:31 node2 kernel: [20840] 101 20840 1889591 120970 406 0 993 postgres Oct 27 07:05:31 node2 kernel: [20851] 101 20851 1891639 122165 408 0 993 postgres Oct 27 07:05:31 node2 kernel: [20955] 101 20955 1889591 120056 405 0 993 postgres Oct 27 07:05:31 node2 kernel: [21117] 101 21117 1891639 121217 405 0 993 postgres Oct 27 07:05:31 node2 kernel: [21328] 101 21328 1891639 120761 404 0 993 postgres Oct 27 07:05:31 node2 kernel: [21332] 101 21332 1889591 119904 402 0 993 postgres Oct 27 07:05:31 node2 kernel: [21451] 101 21451 1891639 120435 403 0 993 postgres Oct 27 07:05:31 node2 kernel: [21988] 101 21988 1889591 121723 407 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 2456] 101 2456 1889592 120006 406 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 2456] 101 2456 1889592 120006 406 0 993 postgres Oct 27 07:05:31 node2 kernel: [32261] 101 32261 1878518 100278 365 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 3352] 101 3352 1926434 149555 459 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 3893] 101 3893 1926436 149552 459 0 993 postgres Oct 27 07:05:31 node2 kernel: [15217] 101 15217 1880573 101541 370 0 993 postgres Oct 27 07:05:31 node2 kernel: [15246] 101 15246 1879605 101892 368 0 993 postgres Oct 27 07:05:31 node2 kernel: [14774] 101 14774 1773892 26263 219 0 993 postgres Oct 27 07:05:31 node2 kernel: [15904] 101 15904 1773886 26721 220 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 9266] 101 9266 1744448 15139 468 0 993 postgres Oct 27 07:05:31 node2 kernel: [15536] 101 15536 1800641 46553 261 0 993 postgres Oct 27 07:05:31 node2 kernel: [21552] 101 21552 1918253 139792 444 0 993 postgres Oct 27 07:05:31 node2 kernel: [22239] 101 22239 1769356 21754 210 0 993 postgres Oct 27 07:05:31 node2 kernel: [13344] 101 13344 1767312 19434 206 0 993 postgres Oct 27 07:05:31 node2 kernel: [27568] 101 27568 1743509 11079 327 0 993 postgres Oct 27 07:05:31 node2 kernel: [27754] 101 27754 1743483 10910 313 0 993 postgres Oct 27 07:05:31 node2 kernel: [28036] 101 28036 1743550 12152 385 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 7772] 101 7772 1743520 11171 332 0 993 postgres Oct 27 07:05:31 node2 kernel: [20855] 101 20855 1743564 13723 422 0 993 postgres Oct 27 07:05:31 node2 kernel: [30883] 101 30883 1743518 11180 333 0 993 postgres Oct 27 07:05:31 node2 kernel: [31463] 101 31463 1740560 3099 137 0 993 postgres Oct 27 07:05:31 node2 kernel: [31466] 101 31466 1740560 3084 137 0 993 postgres Oct 27 07:05:31 node2 kernel: [31475] 101 31475 1740560 3101 137 0 993 postgres Oct 27 07:05:31 node2 kernel: [31485] 101 31485 1740560 3080 137 0 993 postgres Oct 27 07:05:31 node2 kernel: [31513] 101 31513 1740560 3082 137 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 3321] 101 3321 1753416 13629 164 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 3325] 101 3325 1753416 13756 164 0 993 postgres Oct 27 07:05:31 node2 kernel: [16159] 101 16159 1743331 6323 310 0 993 postgres Oct 27 07:05:31 node2 kernel: [32368] 101 32368 1743330 6156 311 0 993 postgres Oct 27 07:05:31 node2 kernel: [ 1528] 101 1528 2009656 24435 346 0 993 postgres -- more lines... Oct 27 07:05:31 node2 kernel: Memory cgroup out of memory: Kill process 19609 (postgres) score 1526 or sacrifice child Oct 27 07:05:31 node2 kernel: Killed process 19609 (postgres), UID 101, total-vm:23637472kB, anon-rss:16612848kB, file-rss:19156kB, shmem-rss:824248kB Oct 27 07:05:32 node2 etcd: 2023-10-27 07:05:32.607769 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/default/kubernetes\" " with result "$ Oct 27 07:05:34 node2 kubelet: I1027 07:05:34.624970 24409 kubelet.go:1926] SyncLoop (UPDATE, "api"): "postgres-cluster-0_postgres(5984f099-30b0-4506-873$ Oct 27 07:05:38 node2 kubelet: I1027 07:05:38.823153 24409 setters.go:86] Using node IP: "x.x.x.x" Oct 27 07:05:44 node2 kubelet: I1027 07:05:44.011520 24409 container_manager_linux.go:490] [ContainerManager]: Discovered runtime cgroups name: /systemd/system.slice Oct 27 07:05:44 node2 kubelet: I1027 07:05:44.826318 24409 kubelet.go:1926] SyncLoop (UPDATE, "api"): "postgres-cluster-0_postgres(5984f099-30b0-4506-873$ Oct 27 07:05:48 node2 kubelet: I1027 07:05:48.915249 24409 setters.go:86] Using node IP: "x.x.x.x" Oct 27 07:05:55 node2 kubelet: I1027 07:05:55.104931 24409 kubelet.go:1926] SyncLoop (UPDATE, "api"): "postgres-cluster-0_postgres(5984f099-30b0-4506-873$ Oct 27 07:05:58 node2 kubelet: I1027 07:05:58.997936 24409 setters.go:86] Using node IP: "x.x.x.x" Oct 27 07:06:01 node2 systemd: Started Session 332365 of user root. Oct 27 07:06:06 node2 kubelet: I1027 07:06:06.133210 24409 kubelet.go:1926] SyncLoop (UPDATE, "api"): "postgres-cluster-0_postgres(5984f099-30b0-4506-873$ Oct 27 07:06:08 node2 kubelet: I1027 07:06:08.029905 24409 kubelet_getters.go:178] "Pod status updated" pod="kube-system/kube-scheduler-node2" status=Running Oct 27 07:06:08 node2 kubelet: I1027 07:06:08.029966 24409 kubelet_getters.go:178] "Pod status updated" pod="kube-system/kube-apiserver-node2" status=Running Oct 27 07:06:08 node2 kubelet: I1027 07:06:08.029996 24409 kubelet_getters.go:178] "Pod status updated" pod="kube-system/kube-controller-manager-node2" status=Ru$ Oct 27 07:06:08 node2 kubelet: E1027 07:06:08.874227 24409 remote_runtime.go:392] ExecSync 28f30220f15ec2a26071c2175f24f77d1b20870fcb3c24bc659d824223d93deb '/usr/bin$ Oct 27 07:06:08 node2 kubelet: I1027 07:06:08.874326 24409 prober.go:117] Readiness probe for "calico-kube-controllers-7c5b64bf96-q6rjm_kube-system(14519a12-7721-43b$ Oct 27 07:06:09 node2 kubelet: I1027 07:06:09.090193 24409 setters.go:86] Using node IP: "x.x.x.x" Oct 27 07:06:09 node2 etcd: 2023-10-27 07:06:09.223921 I | mvcc: store.index: compact 76535976 Oct 27 07:06:09 node2 etcd: 2023-10-27 07:06:09.253047 I | mvcc: finished scheduled compaction at 76535976 (took 28.190376ms) Oct 27 07:06:15 node2 kubelet: I1027 07:06:15.232655 24409 kubelet.go:1926] SyncLoop (UPDATE, "api"): "postgres-cluster-0_postgres(5984f099-30b0-4506-873$ Oct 27 07:06:19 node2 kubelet: W1027 07:06:19.053762 24409 kubelet_pods.go:880] Unable to retrieve pull secret /registrypullsecret for /-arcmana$ Oct 27 07:06:19 node2 kubelet: I1027 07:06:19.198356 24409 setters.go:86] Using node IP: "x.x.x.x" Oct 27 07:06:25 node2 kubelet: I1027 07:06:25.132514 24409 kubelet.go:1926] SyncLoop (UPDATE, "api"): "postgres-cluster-0_postgres(5984f099-30b0-4506-873$ Oct 27 07:06:29 node2 kubelet: I1027 07:06:29.279555 24409 setters.go:86] Using node IP: "x.x.x.x" Oct 27 07:06:34 node2 kubelet: I1027 07:06:34.596501 24409 kubelet.go:1926] SyncLoop (UPDATE, "api"): "postgres-cluster-0_postgres(5984f099-30b0-4506-873$ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.005277 W | etcdserver: read-only range request "key:\"/registry/ingress/\" range_end:\"/registry/ingress0\" count_only:$ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.005533 W | etcdserver: request "header:<ID:1981451119914121853 username:\"etcd-node-node1\" auth_revision:1 > txn:<$ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.005961 W | etcdserver: read-only range request "key:\"/registry/volumeattachments/\" range_end:\"/registry/volumeattach$ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.005996 W | etcdserver: read-only range request "key:\"/registry/crd.projectcalico.org/ipamblocks/\" range_end:\"/regist$ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.006012 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "rang$ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.006092 W | etcdserver: read-only range request "key:\"/registry/namespaces/kube-system\" " with result "range_response_$ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.307799 W | etcdserver: read-only range request "key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/\" range$ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.308862 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" $ Oct 27 07:06:36 node2 etcd: 2023-10-27 07:06:36.308906 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "r |
