1. extensions shared_preload_libraries = 'orafce,pgaudit,pg_cron,pg_stat_statements,set_user' 2. psql can not login now ,it hang there too, so can not check anything from pg_stats_* views 3. one main app user and 2 schemas ,no long running transactions . 4. we use /pgdata , it's on xfs , lvm/vg RHEL8.4 ,it's a shared storage, no use root filesystem. /dev/mapper/pgdatavg-pgdatalv 500G 230G 271G 46% /pgdata /dev/mapper/pgdatavg-pgarchivelv 190G 1.5G 189G 1% /pgarchive /dev/mapper/pgdatavg-pgwallv 100G 34G 67G 34% /pgwal Regards, James -----Original Message----- From: Justin Pryzby <pryzby@xxxxxxxxxxxxx> Sent: Sunday, May 29, 2022 11:02 PM To: James Pang (chaolpan) <chaolpan@xxxxxxxxx> Cc: pgsql-performance@xxxxxxxxxxxxxxxxxxxx Subject: Re: postgres backend process hang on " D " state On Sun, May 29, 2022 at 01:20:12PM +0000, James Pang (chaolpan) wrote: > We have a performance test on Postgresql 13.4 on RHEL8.4 , just after connection storm in ( 952 new connections coming in 1 minute), a lot of backends start on " D " state, and when more sessions got disconnected, they do not exit successfully, instead became "defunct". No errors from postgresql.log , just after the connection storm, some pg_cron workers can not started either. The server is a Virtual machine and no IO hang (actually) IO load is very low. Could be a postgresql bug or an OS abnormal behavior? What settings have you set ? https://wiki.postgresql.org/wiki/Server_Configuration What extensions do you have loaded? \dx Send the output of SELECT * FROM pg_stat_activity either as an attachment or in \x mode? What is your data dir ? Is it on the VM's root filesystem or something else ? Show the output of "mount". Are there any kernel messages in /var/log/messages or `dmesg` ? How many relations are in your schema ? Are you using temp tables ? Long-running transactions ? -- Justin