I did more testing to find out a more specific test case. It seems that
indeed encrypted swap on LVM logical volume is needed. Swap on
unencrypted LVM volume, encrypted swap on raw partition or no swap at
all is not enough. On those cases, OOM killer starts after both RAM and
swap (if any) has been exhausted and after that it's possible to recover
if essential processes did not get killed. The same happens with either
tmpfs, SysV shm and just malloc().
However, in case swap is on encrypted LVM volume, the system becomes
very unresponsive after RAM (not even swap yet) is filled with either
tmpfs or SysV shm. It's possible to use SysRq and switch VTs (but it
happens slowly). But bash does not respond and the cursor can stop
blinking for a while. OOM killer is not triggered. Manual invocation of
OOM killer with SysRq kills the bad process, but the system never
recovers. Exhausting the RAM+swap with malloc() does not trigger this.
Here's the entry for swap in /etc/crypttab:
cswap /dev/mapper/levy-swap /dev/urandom
cipher=aes-xts-plain64,size=256,hash=sha1,swap
/dev/mapper/levy-swap is a LVM volume on SSD with the same size as RAM
(16GB).
I tested this with Debian kernel 5.6.0-1-amd64.
-Topi
