Dear Linux folks,
On a 1 TB RAM compute server with Linux 5.10.24 and memory
overcommitting disabled, we ran into a situation where processes like
SSH couldn’t allocate memory anymore.
$ more /proc/cmdline
BOOT_IMAGE=/boot/bzImage-5.10.24.mx64.375 root=LABEL=root ro
crashkernel=256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd
audit=0 random.trust_cpu=on
2021-06-03T22:00:28+02:00 godsavethequeen sshd[89163]:
pam_systemd(sshd:session): Failed to create session: Unit
session-25654.scope not found.
2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: error:
do_exec_no_pty: fork: Cannot allocate memory
2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]:
pam_unix(sshd:session): session closed for user root
2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: fork:
Cannot allocate memory
2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error:
ssh_msg_send: write: Broken pipe
2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error:
send_rexec_state: ssh_msg_send failed
$ free -h
total used free shared buff/cache
available
Mem: 1.0T 606G 2.6G 2.2M 395G
391G
Swap: 0B 0B 0B
Looking at this, I would have expected, that the pages(?) in buff/cache
would be moved/deleted to make memory available.
Looking at `/proc/meminfo` (attached):
MemTotal: 1052411824 kB
MemFree: 2709976 kB
MemAvailable: 410847908 kB
[…]
CommitLimit: 1052411824 kB
Committed_AS: 1052455260 kB
[…]
Committed_AS is greater than the commit limit (total memory).
Is such behavior expected?
Kind regards,
Paul
MemTotal: 1052411824 kB
MemFree: 2709976 kB
MemAvailable: 410847908 kB
Buffers: 3212 kB
Cached: 411083788 kB
SwapCached: 0 kB
Active: 303175824 kB
Inactive: 740080100 kB
Active(anon): 1448 kB
Inactive(anon): 632169724 kB
Active(file): 303174376 kB
Inactive(file): 107910376 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 632161948 kB
Mapped: 508880 kB
Shmem: 2248 kB
KReclaimable: 2357416 kB
Slab: 3119240 kB
SReclaimable: 2357416 kB
SUnreclaim: 761824 kB
KernelStack: 50512 kB
PageTables: 1715372 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1052411824 kB
Committed_AS: 1052455260 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 480508 kB
VmallocChunk: 0 kB
Percpu: 460800 kB
AnonHugePages: 509384704 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 5995344 kB
DirectMap2M: 483590144 kB
DirectMap1G: 579862528 kB
