|
Hi, We have a performance test on Postgresql 13.4 on RHEL8.4 , just after connection storm in ( 952 new connections coming in 1 minute), a lot of backends start on “ D “ state, and when more sessions got disconnected, they do not exit
successfully, instead became “defunct”. No errors from postgresql.log , just after the connection storm, some pg_cron workers can not started either. The server is a Virtual machine and no IO hang (actually) IO load is very low. Could be a postgresql
bug or an OS abnormal behavior? top - 13:18:02 up 4 days, 6:59, 6 users, load average: 308.68, 307.93, 307.40 Tasks: 1690 total, 1 running, 853 sleeping, 0 stopped, 836 zombie %Cpu(s): 0.1 us, 0.8 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 128657.6 total, 1188.7 free, 52921.5 used, 74547.4 buff/cache MiB Swap: 3072.0 total, 3066.7 free, 5.3 used. 74757.3 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1070325 postgres 20 0 45.6g 15356 11776 D 0.5 0.0 19:21.37 postgres: xuser xdb 192.x.x.132(33318) BIND 1070328 postgres 20 0 45.6g 16120 11660 D 0.5 0.0 19:21.96 postgres: xuser xdb 192.x.x.121(34372) BIND 1070329 postgres 20 0 45.6g 15380 11872 D 0.5 0.0 19:20.90 postgres: xuser xdb 192.x.x.126(41316) BIND 1070397 postgres 20 0 45.6g 14804 11604 D 0.5 0.0 19:23.57 postgres: xuser xdb 192.x.x.132(33324) BIND 1070434 postgres 20 0 45.6g 14928 11812 D 0.5 0.0 19:21.14 postgres: xuser xdb 192.x.x.129(57298) BIND 1070480 postgres 20 0 45.6g 14612 11660 D 0.5 0.0 19:19.88 postgres: xuser xdb 192.x.x.127(52424) BIND 1070508 postgres 20 0 45.6g 14928 11812 D 0.5 0.0 19:20.48 postgres: xuser xdb 192.x.x.127(52428) BIND 1070523 postgres 20 0 45.6g 14544 11716 D 0.5 0.0 19:22.53 postgres: xuser xdb 192.x.x.130(33678) BIND 1070647 postgres 20 0 45.6g 14444 11660 D 0.5 0.0 19:24.36 postgres: xuser xdb 192.x.x.129(57316) BIND 1070648 postgres 20 0 45.6g 14352 11524 D 0.5 0.0 19:22.86 postgres: xuser xdb 192.x.x.133(48796) BIND 1070676 postgres 20 0 45.6g 14456 11660 D 0.5 0.0 19:21.92 postgres: xuser xdb 192.x.x.128(54614) BIND 1070724 postgres 20 0 45.6g 14352 11524 D 0.5 0.0 19:20.90 postgres: xuser xdb 192.x.x.126(41370) BIND 1070739 postgres 20 0 45.6g 14008 11412 D 0.5 0.0 19:22.69 postgres: xuser xdb 192.x.x.123(56164) BIND 1070786 postgres 20 0 45.6g 14352 11524 D 0.5 0.0 19:22.51 postgres: xuser xdb 192.x.x.121(34428) BIND 1070801 postgres 20 0 45.6g 13240 10688 D 0.5 0.0 19:22.19 postgres: xuser xdb 192.x.x.126(41382) BIND 1070815 postgres 20 0 45.6g 13240 10688 D 0.5 0.0 19:21.36 postgres: xuser xdb 192.x.x.53(55950) BIND 1070830 postgres 20 0 45.6g 13240 10688 D 0.5 0.0 19:23.80 postgres: xuser xdb 192.x.x.131(41704) BIND 1070841 postgres 20 0 45.6g 13304 10744 D 0.5 0.0 19:24.25 postgres: xuser xdb 192.x.x.131(41706) BIND 1070884 postgres 20 0 45.6g 13264 10688 D 0.5 0.0 19:20.61 postgres: xuser xdb 192.x.x.122(33734) BIND 1070903 postgres 20 0 45.6g 14456 11660 D 0.5 0.0 19:23.43 postgres: xuser xdb 192.x.x.132(33384) BIND 1070915 postgres 20 0 45.5g 7280 5372 D 0.5 0.0 19:21.20 postgres: xuser xdb 192.x.x.129(57350) initializing 1070941 postgres 20 0 45.5g 7280 5372 D 0.5 0.0 19:20.58 postgres: xuser xdb 192.x.x.124(35934) initializing 1070944 postgres 20 0 45.5g 7280 5372 D 0.5 0.0 19:18.57 postgres: xuser xdb 192.x.x.50(58964) initializing 1070963 postgres 20 0 45.5g 7280 5372 D 0.5 0.0 19:21.98 postgres: xuser xdb 192.x.x.132(33362) initializing 1070974 postgres 20 0 45.5g 7280 5372 D 0.5 0.0 19:22.76 postgres: xuser xdb 192.x.x.54(56774) initializing 1070986 postgres 20 0 45.5g 7284 5372 D 0.5 0.0 19:21.89 postgres: xuser xdb 192.x.x.132(33394) initializing … postgres 1071160 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071161 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071162 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071163 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071164 1951 0 May27 ? 00:00:03 [postmaster] <defunct> postgres 1071167 1951 0 May27 ? 00:00:21 [postmaster] <defunct> postgres 1071168 1951 0 May27 ? 00:00:03 [postmaster] <defunct> postgres 1071170 1951 0 May27 ? 00:00:03 [postmaster] <defunct> postgres 1071171 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071174 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071175 1951 0 May27 ? 00:00:03 [postmaster] <defunct> postgres 1071176 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071179 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071181 1951 0 May27 ? 00:00:03 [postmaster] <defunct> postgres 1071184 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071185 1951 0 May27 ? 00:00:02 [postmaster] <defunct> postgres 1071187 1951 0 May27 ? 00:00:03 [postmaster] <defunct> avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.16 0.00 0.00 99.84 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Thanks, James |
