The kernel seemed to replace the cgroups memory "charging" mechanism in 3.3 with a more efficient implementation [1], but we think it may be broken under Xen virtualization and load. We do not see any issue in Linux 3.2 and earlier. We have documented panics for Fedora kernels 3.3.4-5.fc17.x86_64, 3.3.5-2.fc16.x86_64, and 3.6.10-2.fc16.x86_64 but *not* on Fedora kernels 3.1.0-7.fc16.x86_64 or 3.2.6-3.fc16.x86_64. Many of our services use MemoryLimit= and similar systemd options that create a memory cgroup for the service. This correlates with kernel panics under the following call path (full listing here [2]): [20488075.457394] [<ffffffff811825e7>] ? mem_cgroup_charge_statistics+0x17/0x60 [20488075.457403] [<ffffffff81184ade>] __mem_cgroup_uncharge_common+0xfe/0x330 [20488075.457410] [<ffffffff8100632d>] ? xen_pte_val+0x1d/0x40 [20488075.457417] [<ffffffff81188457>] mem_cgroup_uncharge_page+0x37/0x40 [20488075.457424] [<ffffffff8115e6d1>] page_remove_rmap+0xb1/0x140 It culminates in this failure: [20488075.457183] kernel BUG at arch/x86/mm/fault.c:396! [20488075.457189] invalid opcode: 0000 [#1] SMP There are also reports of similar failures [3] unrelated to systemd use and on non-Fedora kernels. It appears to be an issue with re-attributing the charge for a page to a different cgroup. Any ideas why we would be seeing this with Linux 3.3+? I can generally reproduce the issue (often minutes after booting) on any heavily loaded machine in order to collect any additional data to help troubleshooting. [1] https://lwn.net/Articles/443241/ [2] https://gist.github.com/raw/70afc901a73e427a0a71 [3] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1073238/comments/6 -- David Strauss | david@xxxxxxxxxxxxxxxx | +1 512 577 5827 [mobile] -- To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
