On Wednesday 09 of November 2011, Balbir Singh wrote: > 2011/11/10 Arkadiusz Miśkiewicz <arekm@xxxxxxxx>: > > Hi, > > > > I have a machine with 6GB of ram and a cgroup for apache processes > > limited to memory.limit_in_bytes = "5100M"; > > memory.soft_limit_in_bytes = "5000M"; > > Can you please send the output of memory.stat? Right now there is small number of users using this server (night here) so: # cat /dev/cgroup/memory/somecgroup/httpd/memory.stat cache 511426560 rss 2637795328 mapped_file 2129920 pgpgin 32233755 pgpgout 31509870 inactive_anon 720896 active_anon 2637271040 inactive_file 255959040 active_file 255234048 unevictable 0 hierarchical_memory_limit 5347737600 total_cache 511426560 total_rss 2637795328 total_mapped_file 2129920 total_pgpgin 32233755 total_pgpgout 31509870 total_inactive_anon 720896 total_active_anon 2637271040 total_inactive_file 255959040 total_active_file 255234048 total_unevictable 0 > > > Unfortunately when apache processes ate all ram assigned to their cgroup > > load on whole machine jumps the roof. > > > > cgroup aware OOM kicks in, kills one process and that doesn't help. > > > > If I'm fast enough I notice and then apache processes require tons of > > kill -9 (I'm doing "killall -9 apache" in a while (true) loop for > > 20-30s) to get killed (and that not always succeeds - sometimes I'm > > unable to kill these and I'm just doing sysrq u, s, b after few > > minutes.. if I'm lucky. Sometimes I cannot do any command). > > > > This all happens on 2.6.38.8 kernel. > > http://ixion.pld-linux.org/~arekm/cgroup-eaten-memory-failure-1.txt > > for kernel log. It ends with reboot of the machine. > > From the logs > > "Nov 9 20:48:53 tm2 kernel: [18300.349106] Task in /somecgroup/httpd > killed as a result of limit of /somecgroup/httpd > Nov 9 20:48:53 tm2 kernel: [18300.349110] memory: usage 5222400kB, > limit 5222400kB, failcnt 282869 > Nov 9 20:48:53 tm2 kernel: [18300.349113] memory+swap: usage 0kB, > limit 9007199254740991kB, failcnt 0" > > It seems like you've disabled swap, is that correct? Yes, this machine has no swap but the kernel has # zcat /proc/config.gz |grep CGROUP_MEM CONFIG_CGROUP_MEM_RES_CTLR=y CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y # CONFIG_CGROUP_MEM_RES_CTLR_SWAP_ENABLED is not set + ctlr swap wasn't enabled at boot. > Balbir ps. vger.kernel.org doesn't like my address (no idea why and postmaster@ is silent), so likely my reply won't reach cgroup@ -- Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ Containers mailing list Containers@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linuxfoundation.org/mailman/listinfo/containers
