> On Mar 13, 2023, at 10:37 AM, Joe Conway <mail@xxxxxxxxxxxxx> wrote: > > On 3/13/23 13:55, Israel Brewster wrote: >> 1) They reference a “Memory cgroup out of memory”, which refers back >> to the opening comment on Joe Conway’s message - this would imply to >> me that I *AM* running with a cgroup memory.limit set. Not sure how >> that changes things? > > cgroup memory limit is enforced regardless of the actual host level memory pressure. As an example, if your host VM has 128 GB of memory, but your cgroup memory limit is 512MB, you will get an OOM kill when the sum memory usage of all of your postgres processes (and anything else sharing the same cgroup) exceeds 512 MB, even if the host VM has nothing else going on consuming memory. > > You can check if a memory is set by reading the corresponding virtual file, e.g: > > 8<------------------- > # cat /sys/fs/cgroup/memory/system.slice/postgresql.service/memory.limit_in_bytes > 9223372036854710272 > 8<------------------- > > A few notes: > 1/ The specific path to memory.limit_in_bytes might vary, but this example is the default for the RHEL 8 postgresql 10 RPM. Not finding that file specifically (this is probably too much info, but…): root@novarupta:~# ls /sys/fs/cgroup/system.slice/ -.mount cgroup.threads dev-hugepages.mount memory.events.local memory.swap.events proc-diskstats.mount ssh.service system-postgresql.slice systemd-resolved.service accounts-daemon.service cgroup.type dev-lxc-console.mount memory.high memory.swap.high proc-loadavg.mount sys-devices-system-cpu-online.mount systemd-initctl.socket systemd-sysctl.service cgroup.controllers console-getty.service dev-lxc-tty1.mount memory.low memory.swap.max proc-meminfo.mount sys-devices-virtual-net.mount systemd-journal-flush.service systemd-sysusers.service cgroup.events console-setup.service dev-lxc-tty2.mount memory.max networkd-dispatcher.service proc-stat.mount sys-fs-fuse-connections.mount systemd-journald-audit.socket systemd-tmpfiles-setup-dev.service cgroup.freeze cpu.pressure dev-mqueue.mount memory.min pids.current proc-swaps.mount sys-kernel-debug.mount systemd-journald-dev-log.socket systemd-tmpfiles-setup.service cgroup.max.depth cpu.stat dev-ptmx.mount memory.numa_stat pids.events proc-sys-kernel-random-boot_id.mount syslog.socket systemd-journald.service systemd-update-utmp.service cgroup.max.descendants cron.service io.pressure memory.oom.group pids.max proc-sys-net.mount sysstat.service systemd-journald.socket systemd-user-sessions.service cgroup.procs data.mount keyboard-setup.service memory.pressure pool.mount 'proc-sysrq\x2dtrigger.mount' 'system-container\x2dgetty.slice' systemd-logind.service ufw.service cgroup.stat dbus.service memory.current memory.stat postfix.service proc-uptime.mount system-modprobe.slice systemd-networkd.service uuidd.socket cgroup.subtree_control dbus.socket memory.events memory.swap.current proc-cpuinfo.mount rsyslog.service system-postfix.slice systemd-remount-fs.service root@novarupta:~# ls /sys/fs/cgroup/system.slice/system-postgresql.slice/ cgroup.controllers cgroup.max.depth cgroup.stat cgroup.type io.pressure memory.events.local memory.max memory.oom.group memory.swap.current memory.swap.max pids.max cgroup.events cgroup.max.descendants cgroup.subtree_control cpu.pressure memory.current memory.high memory.min memory.pressure memory.swap.events pids.current postgresql@13-main.service cgroup.freeze cgroup.procs cgroup.threads cpu.stat memory.events memory.low memory.numa_stat memory.stat memory.swap.high pids.events root@novarupta:~# ls /sys/fs/cgroup/system.slice/system-postgresql.slice/postgresql@13-main.service/ cgroup.controllers cgroup.max.depth cgroup.stat cgroup.type io.pressure memory.events.local memory.max memory.oom.group memory.swap.current memory.swap.max pids.max cgroup.events cgroup.max.descendants cgroup.subtree_control cpu.pressure memory.current memory.high memory.min memory.pressure memory.swap.events pids.current cgroup.freeze cgroup.procs cgroup.threads cpu.stat memory.events memory.low memory.numa_stat memory.stat memory.swap.high pids.events > > 2/ The value above, 9223372036854710272 basically means "no limit" has been set. > > 3/ The example assumes cgroup v1. There are very few distro's that enable cgroup v2 by default, and generally I have not seen much cgroup v2 usage in the wild (although I strongly recommend it), but if you are using cgroup v2 the names have changed. You can check by doing: > > 8<--cgroupv2 enabled----------------- > # stat -fc %T /sys/fs/cgroup/ > cgroup2fs > 8<--cgroupv1 enabled----------------- > # stat -fc %T /sys/fs/cgroup/ > tmpfs > 8<------------------- Looks like V2: root@novarupta:~# stat -fc %T /sys/fs/cgroup/ cgroup2fs root@novarupta:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal --- Israel Brewster Software Engineer Alaska Volcano Observatory Geophysical Institute - UAF 2156 Koyukuk Drive Fairbanks AK 99775-7320 Work: 907-474-5172 cell: 907-328-9145 > >> 2) All the entries contain the line "oom_score_adj:0”, which would >> seem to imply that the postmaster, with its -900 score is not being >> directly targeted by the OOM killer. > > Sounds correct > > -- > Joe Conway > PostgreSQL Contributors Team > RDS Open Source Databases > Amazon Web Services: https://aws.amazon.com >
