On 8/1/24 06:54, Lance Yang wrote: > Hi all, > > It's possible to encounter an OOM error if both parent and child cgroups are > configured such that memory.min and memory.max are set to the same values, as > is practice in Kubernetes. Is it a practice in Kubernetes since forever or a recent one? Did it work differently before? > Hmm... I'm not sure that whether this behavior is a bug or an expected aspect of > the kernel design. Hmm I'm not a memcg expert, so I cc'd some. > To reproduce the bug, we can follow these command-based steps: > > 1. Check Kernel Version and OS release: > > ``` > $ uname -r > 6.10.0-rc5+ Were older kernels behaving the same? Anyway memory.min documentations says "Hard memory protection. If the memory usage of a cgroup is within its effective min boundary, the cgroup’s memory won’t be reclaimed under any conditions. If there is no unprotected reclaimable memory available, OOM killer is invoked." So to my non-expert opinion this behavior seems valid. if you set min to the same value as max and then reach the max, you effectively don't allow any reclaim, so the memcg OOM kill is the only option AFAICS? > $ cat /etc/os-release > PRETTY_NAME="Ubuntu 24.04 LTS" > NAME="Ubuntu" > VERSION_ID="24.04" > VERSION="24.04 LTS (Noble Numbat)" > VERSION_CODENAME=noble > ID=ubuntu > ID_LIKE=debian > HOME_URL="<https://www.ubuntu.com/>" > SUPPORT_URL="<https://help.ubuntu.com/>" > BUG_REPORT_URL="<https://bugs.launchpad.net/ubuntu/>" > PRIVACY_POLICY_URL="<https://www.ubuntu.com/legal/terms-and-policies/privacy-policy>" > UBUNTU_CODENAME=noble > LOGO=ubuntu-logo > > ``` > > 2. Navigate to the cgroup v2 filesystem, create a test cgroup, and set memory settings: > > ``` > $ cd /sys/fs/cgroup/ > $ stat -fc %T /sys/fs/cgroup > cgroup2fs > $ mkdir test > $ echo "+memory" > cgroup.subtree_control > $ mkdir test/test-child > $ echo 1073741824 > memory.max > $ echo 1073741824 > memory.min > $ cat memory.max > 1073741824 > $ cat memory.min > 1073741824 > $ cat memory.low > 0 > $ cat memory.high > max > ``` > > 3. Set up and check memory settings in the child cgroup: > > ``` > $ cd test-child > $ echo 1073741824 > memory.max > $ echo 1073741824 > memory.min > $ cat memory.max > 1073741824 > $ cat memory.min > 1073741824 > $ cat memory.low > 0 > $ cat memory.high > max > ``` > > 4. Add process to the child cgroup and verify: > > ``` > $ echo $$ > cgroup.procs > $ cat cgroup.procs > 1131 > 1320 > $ ps -ef|grep 1131 > root 1131 1014 0 10:45 pts/0 00:00:00 -bash > root 1321 1131 99 11:06 pts/0 00:00:00 ps -ef > root 1322 1131 0 11:06 pts/0 00:00:00 grep --color=auto 1131 > ``` > > 5. Attempt to create a large file using dd and observe the process being killed: > > ``` > $ dd if=/dev/zero of=/tmp/2gbfile bs=10M count=200 > Killed > ``` > > 6. Check kernel messages related to the OOM event: > > ``` > $ dmesg > ... > [ 1341.112388] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/test,task_memcg=/test/test-child,task=dd,pid=1324,uid=0 > [ 1341.112418] Memory cgroup out of memory: Killed process 1324 (dd) total-vm:15548kB, anon-rss:10240kB, file-rss:1764kB, shmem-rss:0kB, UID:0 pgtables:76kB oom_score_adj:0 > ``` > > 7. Reduce the `memory.min` setting in the child cgroup and attempt the same large file creation, and then this issue is resolved. > > ``` > # echo 107374182 > memory.min > # dd if=/dev/zero of=/tmp/2gbfile bs=10M count=200 > 200+0 records in > 200+0 records out > 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 1.8713 s, 1.1 GB/s > ``` > > Thanks, > Lance >
