Re: PostgreSQL 12 service failing in Ubuntu 20.04 after a few hours

[Antonis Christodoulou <christan305@xxxxxxxxxxx>]

I just added postgres to the sudoers list, nothing, same issue (slightly different journalctl logs)

This is a machine in the cloud, I can’t disconnect it.

And yes the ps looks like this precisely when I do a fresh restart. I kill all postgres processes and restart:

christan@vultr:~$ for i in `ps -ef | grep postgres | awk '{print $2}'`; do sudo kill $i; done

christan@vultr:~$ sudo systemctl restart postgresql

Then this is the output of me ps:

christan@vultr:~$ ps -ef | grep postgres

postgres 3504795       1  0 05:51 ?        00:00:00 /usr/lib/postgresql/12/bin/postgres -D /var/lib/postgresql/12/main -c config_file=/etc/postgresql/12/main/postgresql.conf

postgres 3504797 3504795  0 05:51 ?        00:00:00 postgres: 12/main: checkpointer   

postgres 3504798 3504795  0 05:51 ?        00:00:00 postgres: 12/main: background writer   

postgres 3504799 3504795  0 05:51 ?        00:00:00 postgres: 12/main: walwriter   

postgres 3504800 3504795  0 05:51 ?        00:00:00 postgres: 12/main: autovacuum launcher   

postgres 3504801 3504795  0 05:51 ?        00:00:00 postgres: 12/main: stats collector   

postgres 3504802 3504795  0 05:51 ?        00:00:00 postgres: 12/main: logical replication launcher   

christan 3504902 3504620  0 05:59 pts/0    00:00:00 grep --color=auto postgres

After a couple of hours, all these go away and I just see that weird ps output… :(

Then this is what I see in journalctl:

Jan 01 21:36:03 vultr.guest systemd[1]: Starting PostgreSQL Cluster 12-main...

Jan 01 21:36:06 vultr.guest systemd[1]: Started PostgreSQL Cluster 12-main.

Jan 01 22:58:44 vultr.guest sudo[3472381]: postgres : TTY=unknown ; PWD=/var/lib/postgresql/12/main ; USER=root ; COMMAND=/usr/sbin/sysctl kernel.nmi_watchdog=0

Jan 01 22:58:44 vultr.guest sudo[3472381]: pam_unix(sudo:session): session opened for user root by (uid=0)

Jan 01 22:58:44 vultr.guest sudo[3472381]: pam_unix(sudo:session): session closed for user root

Jan 01 22:58:57 vultr.guest crontab[3473870]: (postgres) LIST (postgres)

Jan 01 22:58:57 vultr.guest crontab[3473873]: (postgres) LIST (postgres)

…

…

…

Jan 01 22:58:57 vultr.guest crontab[3474021]: (postgres) LIST (postgres)

Jan 01 22:58:57 vultr.guest crontab[3474024]: (postgres) LIST (postgres)

Jan 01 22:58:57 vultr.guest crontab[3474026]: (postgres) REPLACE (postgres)

Jan 01 22:58:57 vultr.guest postgresql@12-main[3474027]: Cluster is not running.

Jan 01 22:58:57 vultr.guest systemd[1]: postgresql@12-main.service: Control process exited, code=exited, status=2/INVALIDARGUMENT

Jan 01 22:58:57 vultr.guest systemd[1]: postgresql@12-main.service: Failed with result 'exit-code'.

I may just install the same way into a dedicated AWS instance, just to see if I’m getting the same behavior there... 

On 1 Jan 2023, at 11:50 PM, Rob Sargent <robjsargent@xxxxxxxxx> wrote:

On 1/1/23 14:48, Adrian Klaver wrote:

On 1/1/23 13:11, Antonis Christodoulou wrote:

Hello Adrian,

No it’s not open, but the database itself has very simple credentials (I am just starting with PostgreSQL). What’s weird about the logs?

Not the logs the ps output. I would expect to see something like:

ps -ef | grep postgres
postgres  395382       1  0  2022 ?        00:03:31 /usr/lib/postgresql/14/bin/postgres -D /var/lib/postgresql/14/main -c config_file=/etc/postgresql/14/main/postgresql.conf
postgres  395384  395382  0  2022 ?        00:00:01 postgres: 14/main: checkpointer
postgres  395385  395382  0  2022 ?        00:00:26 postgres: 14/main: background writer
postgres  395386  395382  0  2022 ?        00:00:26 postgres: 14/main: walwriter
postgres  395387  395382  0  2022 ?        00:01:45 postgres: 14/main: autovacuum launcher
postgres  395388  395382  0  2022 ?        00:00:05 postgres: 14/main: archiver last was 0000000100000003000000B9.00000028.backup
postgres  395389  395382  0  2022 ?        00:01:37 postgres: 14/main: stats collector
postgres  395390  395382  0  2022 ?        00:02:24 postgres: 14/main: pg_cron launcher
postgres  395391  395382  0  2022 ?        00:00:01 postgres: 14/main: logical replication launcher

This:

postgres 3342383       1  0  2022 ?        00:00:00 FzXlkULu
postgres 3344758       1 99  2022 ?        3-14:39:11 OElid7Dp
postgres 3419125       1 18 13:57 ?        01:17:03 tracepath

just does not look right.

At the very least disconnect that machine from the internet.  Completely.