On Wed, 9 Nov 2011 23:58:57 +0100 Arkadiusz Miśkiewicz <arekm@xxxxxxxx> wrote: > On Wednesday 09 of November 2011, Balbir Singh wrote: > > 2011/11/10 Arkadiusz Miśkiewicz <arekm@xxxxxxxx>: > > > Hi, > > > > > > I have a machine with 6GB of ram and a cgroup for apache processes > > > limited to memory.limit_in_bytes = "5100M"; > > > memory.soft_limit_in_bytes = "5000M"; > > > > Can you please send the output of memory.stat? > > Right now there is small number of users using this server (night here) so: > > # cat /dev/cgroup/memory/somecgroup/httpd/memory.stat > cache 511426560 > rss 2637795328 > mapped_file 2129920 > pgpgin 32233755 > pgpgout 31509870 > inactive_anon 720896 > active_anon 2637271040 > inactive_file 255959040 > active_file 255234048 > unevictable 0 > hierarchical_memory_limit 5347737600 > total_cache 511426560 > total_rss 2637795328 > total_mapped_file 2129920 > total_pgpgin 32233755 > total_pgpgout 31509870 > total_inactive_anon 720896 > total_active_anon 2637271040 > total_inactive_file 255959040 > total_active_file 255234048 > total_unevictable 0 > > > > > > Unfortunately when apache processes ate all ram assigned to their cgroup > > > load on whole machine jumps the roof. > > > > > > cgroup aware OOM kicks in, kills one process and that doesn't help. > > > > > > If I'm fast enough I notice and then apache processes require tons of > > > kill -9 (I'm doing "killall -9 apache" in a while (true) loop for > > > 20-30s) to get killed (and that not always succeeds - sometimes I'm > > > unable to kill these and I'm just doing sysrq u, s, b after few > > > minutes.. if I'm lucky. Sometimes I cannot do any command). > > > > > > This all happens on 2.6.38.8 kernel. > > > http://ixion.pld-linux.org/~arekm/cgroup-eaten-memory-failure-1.txt > > > for kernel log. It ends with reboot of the machine. > > > > From the logs > > > > "Nov 9 20:48:53 tm2 kernel: [18300.349106] Task in /somecgroup/httpd > > killed as a result of limit of /somecgroup/httpd > > Nov 9 20:48:53 tm2 kernel: [18300.349110] memory: usage 5222400kB, > > limit 5222400kB, failcnt 282869 > > Nov 9 20:48:53 tm2 kernel: [18300.349113] memory+swap: usage 0kB, > > limit 9007199254740991kB, failcnt 0" > > > > It seems like you've disabled swap, is that correct? > > Yes, this machine has no swap but the kernel has > > # zcat /proc/config.gz |grep CGROUP_MEM > CONFIG_CGROUP_MEM_RES_CTLR=y > CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y > # CONFIG_CGROUP_MEM_RES_CTLR_SWAP_ENABLED is not set > > + ctlr swap wasn't enabled at boot. > > > Balbir > > ps. vger.kernel.org doesn't like my address (no idea why and postmaster@ is > silent), so likely my reply won't reach cgroup@ If you can, please try the latest kernels. recent 2 commits commit 79dfdaccd1d5b40ff7cf4a35a0e63696ebb78b4d " memcg: make oom_lock 0 and 1 based rather than counter " commit 1d65f86db14806cf7b1218c7b4ecb8b4db5af27d " mm: preallocate page before lock_page() at filemap COW" are works well against fork-bomb under memcg. In my test, make -j under swapless memcg hangs (or takes long time) to be oom-killed. Thanks, -Kame -- To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
