hi folks
I have some issue its reasons I am currently unaware of on multiple hosts.
Currently I noticed this when trying to access a postgresql database while doing `su postgres`. please note that I do not think that this is related to postgresql, but the kernel.
as soon as I do `su postgres` sometimes, but not always, the su-process responds immediately with `Killed` and does not su to postgres. this does not happen every time, sometimes I can turn into the postgres user. executing anything else in that shell, e.g. psql then results in `Killed` all the time. I may be hitting some system limits or something I am not aware of.
but if there were limits hit, like things the oomkiller would handle, this would be logged to syslog in any case afaik, yet there are absolutely no logs in syslog, dmesg or the journal. since psql is a perl wrapper script, it forks lots of processes and does not make it to a forked postgresql client. so I tried this script: https://pastebin.com/6igR3pmW
as you can observe, even `date` gets killed sometimes. In this short example at line #22
My best guess now is that there is something wrong with the logging there, the kernel is vanilla 5.5.2, the OS is debian stretch. the ulimits are not hit afaik, at least the usual suspects were set to unlimited, yet any log entry should clarify what's going wrong.
also terminating all processes running as user postgres did not help. afaik ulimits can not be the cause then, as there are no processes running as that user. one weird thing: the database server itself, postgresql, can start perfectly fine.
# ps -elf | grep postgres
0 S postgres 680 1 0 80 0 - 42460 - Feb07 ? 00:00:41 /usr/lib/postgresql/9.6/bin/postgres [... some args]
1 S postgres 681 680 0 80 0 - 34755 - Feb07 ? 00:00:02 postgres: logger process
1 S postgres 684 680 0 80 0 - 42518 - Feb07 ? 00:00:17 postgres: checkpointer process
1 S postgres 685 680 0 80 0 - 42460 - Feb07 ? 00:00:14 postgres: writer process
1 S postgres 686 680 0 80 0 - 42460 - Feb07 ? 00:00:15 postgres: wal writer process
1 S postgres 687 680 0 80 0 - 38389 - Feb07 ? 00:00:12 postgres: stats collector process
0 S postgres 680 1 0 80 0 - 42460 - Feb07 ? 00:00:41 /usr/lib/postgresql/9.6/bin/postgres [... some args]
1 S postgres 681 680 0 80 0 - 34755 - Feb07 ? 00:00:02 postgres: logger process
1 S postgres 684 680 0 80 0 - 42518 - Feb07 ? 00:00:17 postgres: checkpointer process
1 S postgres 685 680 0 80 0 - 42460 - Feb07 ? 00:00:14 postgres: writer process
1 S postgres 686 680 0 80 0 - 42460 - Feb07 ? 00:00:15 postgres: wal writer process
1 S postgres 687 680 0 80 0 - 38389 - Feb07 ? 00:00:12 postgres: stats collector process
what am I missing? could this be some issue with the memory management killing processes but not logging things?
Also, should that help: strace psql - https://pastebin.com/7w2muHDc which is being killed as +++ killed by SIGKILL +++
