Hi there, Hopefully I'm sending this to the right place, this is the first time I've reported a kernel bug. I'm roughly following this format here https://www.kernel.org/pub/linux/docs/lkml/reporting-bugs.html. 1. The OOM killer kicks in to kill processes inside a cgroup that has hit its memory limit but sometimes kills a process outside of the cgroup 2. We've encountered an error on Ubuntu 12.04 running on vsphere with kernel linux-image-3.13.0-32-generic as well as linux-image-3.13.0-35-generic which causes the machine to hard lock up. It is completely unresponsive until hard reset. The issue appears to be that the OOM killer kicks in and kills a process outside of the cgroup that has hit its memory limit. Once this has happened the kernel log shows soft lock ups and eventually completely hangs. We can fairly consistently reproduce this by starting some processes inside an LXC container (via docker) that push the container very close to its memory limit. We then have a cron task that starts up inside the container and causes it to OOM. The OOM killer then kicks in and sometimes will trigger this hang. Every time it happens we see this message: [ 2635.212105] Task in / killed as a result of limit of /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 Notice how it kills a task in "/"? Immediately following that we start seeing this repeated: [ 3164.153296] BUG: soft lockup - CPU#0 stuck for 22s! [php5:15940] And then the machine hangs. It looks very similar to http://permalink.gmane.org/gmane.linux.kernel.mm/108113 and http://irc.13thfloor.at/LOG/2014-02/LOG_2014-02-28.txt + http://217.196.41.9/~ard/bertl-is-the-nicest-guy-in-the-world/jimmy.txt (search for "in / killed"). Kernel log excerpt (fetched via serial using GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8 debug ignore_loglevel nomodeset pciehp_force=1 pciehp_poll_mode=1") [ 2395.454778] docker0: port 2(vethdRqyGV) entered forwarding state [ 2410.500706] docker0: port 2(vethdRqyGV) entered forwarding state [ 2634.814382] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 [ 2634.818286] java cpuset=e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 mems_allowed=0 [ 2634.822596] CPU: 1 PID: 15775 Comm: java Not tainted 3.13.0-32-generic #57~precise1-Ubuntu [ 2634.824960] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 2634.828336] ffff8801375a2000 ffff880137f0bbe8 ffffffff81752c9e ffff88013fd0fff0 [ 2634.830662] ffff8800ab6ddfc0 ffff880137f0bc38 ffffffff81748aba ffff880100000000 [ 2634.832972] 000000d08137cd48 ffff880137f0bc38 ffff8800ae5ec7d0 0000000000000000 [ 2634.835253] Call Trace: [ 2634.836418] [<ffffffff81752c9e>] dump_stack+0x46/0x58 [ 2634.838078] [<ffffffff81748aba>] dump_header+0x7e/0xbd [ 2634.839825] [<ffffffff81748b50>] oom_kill_process.part.5+0x57/0x2d4 [ 2634.841768] [<ffffffff810751b5>] ? has_ns_capability_noaudit+0x15/0x20 [ 2634.843776] [<ffffffff8115b499>] ? oom_badness.part.4+0xa9/0x140 [ 2634.845637] [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50 [ 2634.847424] [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0 [ 2634.849413] [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250 [ 2634.851413] [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0 [ 2634.853296] [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90 [ 2634.855197] [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3 [ 2634.857020] [<ffffffff817638d5>] __do_page_fault+0x545/0x570 [ 2634.858815] [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20 [ 2634.860733] [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0 [ 2634.862595] [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70 [ 2634.864476] [<ffffffff8176391a>] do_page_fault+0x1a/0x70 [ 2634.866197] [<ffffffff8175fa88>] page_fault+0x28/0x30 [ 2634.867954] Task in /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 killed as a result of limit of /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 [ 2634.872851] memory: usage 65536kB, limit 65536kB, failcnt 34251 [ 2634.874714] memory+swap: usage 65536kB, limit 65536kB, failcnt 1 [ 2634.876622] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0 [ 2634.878527] Memory cgroup stats for /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4: cache:412KB rss:65124KB rss_huge:0KB mapped_file:4KB writeback:0KB swap:0KB inactive_anon:33392KB active_anon:32052KB inactive_file:88KB active_file:0KB unevictable:0KB [ 2634.888650] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 2634.891018] [15552] 0 15552 12511 732 29 0 0 sshd [ 2634.893373] [15596] 0 15596 5180 276 15 0 0 cron [ 2634.895686] [15731] 0 15731 19971 926 43 0 0 sshd [ 2634.898004] [15735] 1014 15735 19971 395 40 0 0 sshd [ 2634.900305] [15736] 1014 15736 2760 376 10 0 0 bash [ 2634.902588] [15756] 1014 15756 551397 14730 92 0 0 java [ 2634.904853] [15757] 1014 15757 2760 152 10 0 0 bash [ 2634.907080] [15758] 1014 15758 2760 158 10 0 0 bash [ 2634.909316] [15759] 1014 15759 1472 171 7 0 0 tee [ 2634.911495] [15760] 1014 15760 1472 172 8 0 0 tee [ 2634.913689] [15936] 0 15936 11535 338 28 0 0 cron [ 2634.915905] [15937] 0 15937 1102 153 8 0 0 sh [ 2634.918055] [15938] 0 15938 1102 153 8 0 0 maxlifetime [ 2634.920385] [15940] 0 15940 53661 2029 105 0 0 php5 [ 2634.922570] Memory cgroup out of memory: Kill process 15919 (java) score 904 or sacrifice child [ 2634.924952] Killed process 15758 (bash) total-vm:11040kB, anon-rss:216kB, file-rss:416kB [ 2634.937740] php5 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 [ 2634.939903] php5 cpuset=e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 mems_allowed=0 [ 2634.943244] CPU: 0 PID: 15940 Comm: php5 Not tainted 3.13.0-32-generic #57~precise1-Ubuntu [ 2634.945628] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 2634.949166] ffff8801375a2000 ffff8800920c5be8 ffffffff81752c9e ffff88013fc0fff0 [ 2634.951526] ffff8800ab1a47d0 ffff8800920c5c38 ffffffff81748aba ffff880000000000 [ 2634.953906] 000000d08137cd48 ffff8800920c5c38 ffff8800ae5ec7d0 0000000000000000 [ 2634.956274] Call Trace: [ 2634.957475] [<ffffffff81752c9e>] dump_stack+0x46/0x58 [ 2634.959212] [<ffffffff81748aba>] dump_header+0x7e/0xbd [ 2634.960979] [<ffffffff81748b50>] oom_kill_process.part.5+0x57/0x2d4 [ 2634.962970] [<ffffffff810751b5>] ? has_ns_capability_noaudit+0x15/0x20 [ 2634.965016] [<ffffffff8115b499>] ? oom_badness.part.4+0xa9/0x140 [ 2634.966913] [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50 [ 2634.968739] [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0 [ 2634.970768] [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250 [ 2634.972788] [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0 [ 2634.974613] [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90 [ 2634.976509] [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3 [ 2634.978233] [<ffffffff817638d5>] __do_page_fault+0x545/0x570 [ 2634.980032] [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20 [ 2634.981886] [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0 [ 2634.983713] [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70 [ 2634.985577] [<ffffffff8176391a>] do_page_fault+0x1a/0x70 [ 2634.987283] [<ffffffff8175fa88>] page_fault+0x28/0x30 [ 2634.988982] Task in /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 killed as a result of limit of /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 [ 2634.993837] memory: usage 65536kB, limit 65536kB, failcnt 34462 [ 2634.995715] memory+swap: usage 65536kB, limit 65536kB, failcnt 1 [ 2634.997667] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0 [ 2634.999596] Memory cgroup stats for /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4: cache:464KB rss:65032KB rss_huge:0KB mapped_file:16KB writeback:0KB swap:0KB inactive_anon:33288KB active_anon:32068KB inactive_file:92KB active_file:12KB unevictable:0KB [ 2635.008960] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 2635.011353] [15552] 0 15552 12511 732 29 0 0 sshd [ 2635.013738] [15596] 0 15596 5180 276 15 0 0 cron [ 2635.016195] [15731] 0 15731 19971 926 43 0 0 sshd [ 2635.018559] [15735] 1014 15735 19971 395 40 0 0 sshd [ 2635.020947] [15736] 1014 15736 2760 376 10 0 0 bash [ 2635.023288] [15756] 1014 15756 551397 14759 92 0 0 java [ 2635.025590] [15757] 1014 15757 2760 152 10 0 0 bash [ 2635.027821] [15759] 1014 15759 1472 171 7 0 0 tee [ 2635.030055] [15760] 1014 15760 1472 172 8 0 0 tee [ 2635.032287] [15936] 0 15936 11535 338 28 0 0 cron [ 2635.034570] [15937] 0 15937 1102 153 8 0 0 sh [ 2635.036809] [15938] 0 15938 1102 153 8 0 0 maxlifetime [ 2635.039211] [15940] 0 15940 53661 2029 105 0 0 php5 [ 2635.041469] Memory cgroup out of memory: Kill process 15919 (java) score 904 or sacrifice child [ 2635.043872] Killed process 15757 (bash) total-vm:11040kB, anon-rss:216kB, file-rss:392kB [ 2635.050025] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 [ 2635.052155] java cpuset=e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 mems_allowed=0 [ 2635.055293] CPU: 0 PID: 15768 Comm: java Not tainted 3.13.0-32-generic #57~precise1-Ubuntu [ 2635.057673] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 2635.061178] ffff8801375a2000 ffff88009212fbe8 ffffffff81752c9e 000000000000b9e4 [ 2635.063548] ffff8800b858dfc0 ffff88009212fc38 ffffffff81748aba ffff880000000000 [ 2635.066421] 000000d08137cd48 ffff88009212fc38 ffff8800ae5ec7d0 0000000000000000 [ 2635.069043] Call Trace: [ 2635.070234] [<ffffffff81752c9e>] dump_stack+0x46/0x58 [ 2635.072018] [<ffffffff81748aba>] dump_header+0x7e/0xbd [ 2635.073804] [<ffffffff81748b50>] oom_kill_process.part.5+0x57/0x2d4 [ 2635.075796] [<ffffffff810751b5>] ? has_ns_capability_noaudit+0x15/0x20 [ 2635.077847] [<ffffffff8115b499>] ? oom_badness.part.4+0xa9/0x140 [ 2635.079754] [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50 [ 2635.081600] [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0 [ 2635.083604] [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250 [ 2635.085649] [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0 [ 2635.087488] [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90 [ 2635.089409] [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3 [ 2635.091155] [<ffffffff817638d5>] __do_page_fault+0x545/0x570 [ 2635.092958] [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20 [ 2635.094819] [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0 [ 2635.096657] [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70 [ 2635.098493] [<ffffffff8176391a>] do_page_fault+0x1a/0x70 [ 2635.100257] [<ffffffff8175fa88>] page_fault+0x28/0x30 [ 2635.101917] Task in /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 killed as a result of limit of /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 [ 2635.106727] memory: usage 65536kB, limit 65536kB, failcnt 34826 [ 2635.108694] memory+swap: usage 65416kB, limit 65536kB, failcnt 1 [ 2635.110609] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0 [ 2635.112573] Memory cgroup stats for /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4: cache:452KB rss:65084KB rss_huge:0KB mapped_file:4KB writeback:0KB swap:0KB inactive_anon:33304KB active_anon:32104KB inactive_file:128KB active_file:0KB unevictable:0KB [ 2635.120953] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 2635.123326] [15552] 0 15552 12511 732 29 0 0 sshd [ 2635.125696] [15596] 0 15596 5180 276 15 0 0 cron [ 2635.128042] [15731] 0 15731 19971 926 43 0 0 sshd [ 2635.130337] [15735] 1014 15735 19971 395 40 0 0 sshd [ 2635.132641] [15736] 1014 15736 2760 376 10 0 0 bash [ 2635.134904] [15756] 1014 15756 551397 14752 92 0 0 java [ 2635.137170] [15759] 1014 15759 1472 171 7 0 0 tee [ 2635.139389] [15760] 1014 15760 1472 172 8 0 0 tee [ 2635.141612] [15936] 0 15936 11535 338 28 0 0 cron [ 2635.143833] [15937] 0 15937 1102 153 8 0 0 sh [ 2635.146024] [15938] 0 15938 1102 153 8 0 0 maxlifetime [ 2635.148378] [15940] 0 15940 55252 2162 108 0 0 php5 [ 2635.150580] Memory cgroup out of memory: Kill process 15919 (java) score 906 or sacrifice child [ 2635.153010] Killed process 15919 (java) total-vm:2205588kB, anon-rss:58444kB, file-rss:564kB [ 2635.160153] php5 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 [ 2635.162238] php5 cpuset=e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 mems_allowed=0 [ 2635.165420] CPU: 0 PID: 15940 Comm: php5 Not tainted 3.13.0-32-generic #57~precise1-Ubuntu [ 2635.167811] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 2635.171392] ffff8801375a2000 ffff8800920c5be8 ffffffff81752c9e 0000000000000007 [ 2635.173904] ffff8800ab1a47d0 ffff8800920c5c38 ffffffff81748aba ffff880000000000 [ 2635.176363] 000000d08137cd48 ffff8800920c5c38 ffff880137f2c7d0 0000000000000000 [ 2635.178731] Call Trace: [ 2635.179924] [<ffffffff81752c9e>] dump_stack+0x46/0x58 [ 2635.181732] [<ffffffff81748aba>] dump_header+0x7e/0xbd [ 2635.183532] [<ffffffff81748b50>] oom_kill_process.part.5+0x57/0x2d4 [ 2635.185606] [<ffffffff810751b5>] ? has_ns_capability_noaudit+0x15/0x20 [ 2635.187661] [<ffffffff8115b499>] ? oom_badness.part.4+0xa9/0x140 [ 2635.189642] [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50 [ 2635.191472] [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0 [ 2635.193567] [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250 [ 2635.195601] [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0 [ 2635.197548] [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90 [ 2635.199548] [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3 [ 2635.201309] [<ffffffff817638d5>] __do_page_fault+0x545/0x570 [ 2635.203099] [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20 [ 2635.205016] [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0 [ 2635.206847] [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70 [ 2635.208699] [<ffffffff8176391a>] do_page_fault+0x1a/0x70 [ 2635.210413] [<ffffffff8175fa88>] page_fault+0x28/0x30 **-> [ 2635.212105] Task in / killed as a result of limit of /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4 [ 2635.215651] memory: usage 5400kB, limit 65536kB, failcnt 34938 [ 2635.217580] memory+swap: usage 5400kB, limit 65536kB, failcnt 1 [ 2635.219498] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0 [ 2635.221488] Memory cgroup stats for /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4: cache:452KB rss:4088KB rss_huge:0KB mapped_file:0KB writeback:0KB swap:0KB inactive_anon:1080KB active_anon:3328KB inactive_file:132KB active_file:0KB unevictable:0KB [ 2635.233441] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 2635.235809] [15552] 0 15552 12511 732 29 0 0 sshd [ 2635.238209] [15596] 0 15596 5180 276 15 0 0 cron [ 2635.240604] [15936] 0 15936 11535 338 28 0 0 cron [ 2635.242918] [15937] 0 15937 1102 153 8 0 0 sh [ 2635.245179] [15938] 0 15938 1102 153 8 0 0 maxlifetime [ 2635.247564] [15940] 0 15940 55252 2162 108 0 0 php5 [ 2635.249819] Memory cgroup out of memory: Kill process 15861 (java) score 918 or sacrifice child [ 2659.884484] BUG: soft lockup - CPU#0 stuck for 23s! [php5:15940] [ 2659.888830] Modules linked in: xt_nat veth ipt_REJECT xt_tcpudp xt_state xt_addrtype iptable_filter xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc aufs psmouse vmwgfx serio_raw ttm drm ppdev vmw_vmci vmw_balloon parport_pc shpchp i2c_piix4 mac_hid lp parport e1000 floppy mptspi mptscsih mptbase vmw_pvscsi vmxnet3 [ 2659.908560] CPU: 0 PID: 15940 Comm: php5 Not tainted 3.13.0-32-generic #57~precise1-Ubuntu [ 2659.912330] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 2659.917958] task: ffff8800ab1a47d0 ti: ffff8800920c4000 task.ti: ffff8800920c4000 [ 2659.921601] RIP: 0010:[<ffffffff81748c29>] [<ffffffff81748c29>] oom_kill_process.part.5+0x130/0x2d4 [ 2659.927119] RSP: 0000:ffff8800920c5c48 EFLAGS: 00000246 [ 2659.930090] RAX: ffff8800a0ffe358 RBX: 0000000000000396 RCX: 0000000000000000 [ 2659.932371] RDX: ffff88013fc0fff0 RSI: ffff88013fc0e3d8 RDI: ffffffff81c06040 [ 2659.934649] RBP: ffff8800920c5ca8 R08: 0000000000000000 R09: ffff8800ba7b1410 [ 2660.093701] R10: 0000000000000773 R11: ffffffff8185b248 R12: ffff880137f2c7d0 [ 2660.097119] R13: 0000000000000005 R14: 0000000000000246 R15: 00000000375a2000 [ 2660.100528] FS: 00007ff405dc2700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 2660.103897] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2660.105903] CR2: 000000000105c188 CR3: 00000000b86e1000 CR4: 00000000000007f0 [ 2660.108262] Stack: [ 2660.109477] ffff8800920c5c58 ffff880137f2c7d0 0000000000004000 ffff8801375a2000 [ 2660.111952] ffff8800920c5ca8 ffff8800a0ffe358 ffff8800ba578000 ffff880137f2c7d0 [ 2660.114437] 0000000000004000 0000000000000000 ffff8801375a2000 0000000000000000 [ 2660.116912] Call Trace: [ 2660.118209] [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50 [ 2660.120166] [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0 [ 2660.122291] [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250 [ 2660.124439] [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0 [ 2660.126415] [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90 [ 2660.128503] [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3 [ 2660.130405] [<ffffffff817638d5>] __do_page_fault+0x545/0x570 [ 2660.132395] [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20 [ 2660.134474] [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0 [ 2660.136605] [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70 [ 2660.138612] [<ffffffff8176391a>] do_page_fault+0x1a/0x70 [ 2660.140483] [<ffffffff8175fa88>] page_fault+0x28/0x30 [ 2660.364234] Code: 4d 8b a4 24 18 03 00 00 49 81 ec 18 03 00 00 49 8d 84 24 18 03 00 00 4c 39 e8 75 9c 49 8b 87 98 03 00 00 48 89 45 c8 4c 8b 7d c8 <49> 81 ef 98 03 00 00 4c 39 fb 0f 85 66 ff ff ff f0 ff 05 00 d4 [ 2687.763079] BUG: soft lockup - CPU#0 stuck for 23s! [php5:15940] [ 2687.765130] Modules linked in: xt_nat veth ipt_REJECT xt_tcpudp xt_state xt_addrtype iptable_filter xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc aufs psmouse vmwgfx serio_raw ttm drm ppdev vmw_vmci vmw_balloon parport_pc shpchp i2c_piix4 mac_hid lp parport e1000 floppy mptspi mptscsih mptbase vmw_pvscsi vmxnet3 [ 2687.778062] CPU: 0 PID: 15940 Comm: php5 Not tainted 3.13.0-32-generic #57~precise1-Ubuntu [ 2687.781178] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 2687.784908] task: ffff8800ab1a47d0 ti: ffff8800920c4000 task.ti: ffff8800920c4000 [ 2687.787208] RIP: 0010:[<ffffffff81748b9f>] [<ffffffff81748b9f>] oom_kill_process.part.5+0xa6/0x2d4 [ 2687.790638] RSP: 0000:ffff8800920c5c48 EFLAGS: 00000202 [ 2687.792506] RAX: ffff8800a0ffe358 RBX: 0000000000000396 RCX: 0000000000000000 [ 2687.794825] RDX: ffff88013fc0fff0 RSI: ffff88013fc0e3d8 RDI: ffffffff81c06040 [ 2687.797074] RBP: ffff8800920c5ca8 R08: 0000000000000000 R09: ffff8800ba7b1410 [ 2687.799314] R10: 0000000000000773 R11: ffffffff8185b248 R12: ffff880137f2c7d0 [ 2687.801565] R13: 0000000000000005 R14: 0000000000000246 R15: 00000000375a2000 [ 2688.028141] FS: 00007ff405dc2700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 2688.031808] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2688.034737] CR2: 000000000105c188 CR3: 00000000b86e1000 CR4: 00000000000007f0 [ 2688.037416] Stack: [ 2688.038594] ffff8800920c5c58 ffff880137f2c7d0 0000000000004000 ffff8801375a2000 [ 2688.041050] ffff8800920c5ca8 ffff8800a0ffe358 ffff8800ba578000 ffff880137f2c7d0 [ 2688.043501] 0000000000004000 0000000000000000 ffff8801375a2000 0000000000000000 [ 2688.045972] Call Trace: [ 2688.047262] [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50 [ 2688.049245] [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0 [ 2688.051383] [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250 [ 2688.053562] [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0 [ 2688.055578] [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90 [ 2688.057675] [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3 [ 2688.059606] [<ffffffff817638d5>] __do_page_fault+0x545/0x570 [ 2688.061588] [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20 [ 2688.063655] [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0 [ 2688.065647] [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70 [ 2688.067645] [<ffffffff8176391a>] do_page_fault+0x1a/0x70 [ 2688.069508] [<ffffffff8175fa88>] page_fault+0x28/0x30 [ 2688.071297] Code: 00 00 45 89 e8 48 c7 c7 08 50 a7 81 31 c0 e8 1c e1 ff ff 4c 89 e7 e8 b1 66 01 00 48 c7 c7 40 60 c0 81 e8 05 63 01 00 48 89 5d a8 <4d> 8b a7 08 03 00 00 4d 8d af 08 03 00 00 49 81 ec 18 03 00 00 [ 2715.786036] BUG: soft lockup - CPU#0 stuck for 22s! [php5:15940] [ 2715.788772] Modules linked in: xt_nat veth ipt_REJECT xt_tcpudp xt_state xt_addrtype iptable_filter xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc aufs psmouse vmwgfx serio_raw ttm drm ppdev vmw_vmci vmw_balloon parport_pc shpchp i2c_piix4 mac_hid lp parport e1000 floppy mptspi mptscsih mptbase vmw_pvscsi vmxnet3 [ 2715.801184] CPU: 0 PID: 15940 Comm: php5 Not tainted 3.13.0-32-generic #57~precise1-Ubuntu [ 2715.803616] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 2715.807270] task: ffff8800ab1a47d0 ti: ffff8800920c4000 task.ti: ffff8800920c4000 [ 2715.809571] RIP: 0010:[<ffffffff81748c30>] [<ffffffff81748c30>] oom_kill_process.part.5+0x137/0x2d4 [ 2715.812972] RSP: 0000:ffff8800920c5c48 EFLAGS: 00000286 [ 2715.814858] RAX: ffff8800a0ffe358 RBX: 0000000000000396 RCX: 0000000000000000 [ 2715.817102] RDX: ffff88013fc0fff0 RSI: ffff88013fc0e3d8 RDI: ffffffff81c06040 [ 2715.819350] RBP: ffff8800920c5ca8 R08: 0000000000000000 R09: ffff8800ba7b1410 [ 2715.821619] R10: 0000000000000773 R11: ffffffff8185b248 R12: ffff880137f2c7d0 [ 2715.823852] R13: 0000000000000005 R14: 0000000000000246 R15: 00000000375a2000 [ 2715.826095] FS: 00007ff405dc2700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 2716.058770] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2716.060776] CR2: 000000000105c188 CR3: 00000000b86e1000 CR4: 00000000000007f0 [ 2716.063102] Stack: [ 2716.064279] ffff8800920c5c58 ffff880137f2c7d0 0000000000004000 ffff8801375a2000 [ 2716.066731] ffff8800920c5ca8 ffff8800a0ffe358 ffff8800ba578000 ffff880137f2c7d0 [ 2716.069221] 0000000000004000 0000000000000000 ffff8801375a2000 0000000000000000 [ 2716.071694] Call Trace: [ 2716.072983] [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50 [ 2716.074951] [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0 [ 2716.077116] [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250 [ 2716.079305] [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0 [ 2716.081322] [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90 [ 2716.083443] [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3 [ 2716.085375] [<ffffffff817638d5>] __do_page_fault+0x545/0x570 [ 2716.087364] [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20 [ 2716.089458] [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0 [ 2716.091451] [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70 [ 2716.093447] [<ffffffff8176391a>] do_page_fault+0x1a/0x70 [ 2716.095312] [<ffffffff8175fa88>] page_fault+0x28/0x30 [ 2716.097125] Code: 00 49 81 ec 18 03 00 00 49 8d 84 24 18 03 00 00 4c 39 e8 75 9c 49 8b 87 98 03 00 00 48 89 45 c8 4c 8b 7d c8 49 81 ef 98 03 00 00 <4c> 39 fb 0f 85 66 ff ff ff f0 ff 05 00 d4 4b 00 48 8b 7d a8 e8 [ 2743.821209] BUG: soft lockup - CPU#0 stuck for 22s! [php5:15940] [ 2743.823189] Modules linked in: xt_nat veth ipt_REJECT xt_tcpudp xt_state xt_addrtype iptable_filter xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc aufs psmouse vmwgfx serio_raw ttm drm ppdev vmw_vmci vmw_balloon parport_pc shpchp i2c_piix4 mac_hid lp parport e1000 floppy mptspi mptscsih mptbase vmw_pvscsi vmxnet3 [ 2743.835431] CPU: 0 PID: 15940 Comm: php5 Not tainted 3.13.0-32-generic #57~precise1-Ubuntu [ 2743.837879] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 [ 2743.841550] task: ffff8800ab1a47d0 ti: ffff8800920c4000 task.ti: ffff8800920c4000 [ 2743.843848] RIP: 0010:[<ffffffff81748ba6>] [<ffffffff81748ba6>] oom_kill_process.part.5+0xad/0x2d4 [ 2743.847267] RSP: 0000:ffff8800920c5c48 EFLAGS: 00000202 [ 2743.849129] RAX: ffff8800a0ffe358 RBX: 0000000000000396 RCX: 0000000000000000 [ 2743.851396] RDX: ffff88013fc0fff0 RSI: ffff88013fc0e3d8 RDI: ffffffff81c06040 [ 2743.974848] RBP: ffff8800920c5ca8 R08: 0000000000000000 R09: ffff8800ba7b1410 [ 2743.978217] R10: 0000000000000773 R11: ffffffff8185b248 R12: ffff880137f2c7d0 [ 2743.981510] R13: 0000000000000005 R14: 0000000000000246 R15: 00000000375a2000 [ 2743.983737] FS: 00007ff405dc2700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 2743.986172] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2743.988121] CR2: 000000000105c188 CR3: 00000000b86e1000 CR4: 00000000000007f0 [ 2743.990371] Stack: [ 2743.991555] ffff8800920c5c58 ffff880137f2c7d0 0000000000004000 ffff8801375a2000 [ 2743.994003] ffff8800920c5ca8 ffff8800a0ffe358 ffff8800ba578000 ffff880137f2c7d0 [ 2743.996448] 0000000000004000 0000000000000000 ffff8801375a2000 0000000000000000 [ 2743.998918] Call Trace: [ 2744.000205] [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50 [ 2744.002173] [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0 [ 2744.004317] [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250 [ 2744.006492] [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0 [ 2744.008507] [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90 [ 2744.010597] [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3 [ 2744.012529] [<ffffffff817638d5>] __do_page_fault+0x545/0x570 [ 2744.014536] [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20 [ 2744.016601] [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0 [ 2744.018597] [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70 [ 2744.020595] [<ffffffff8176391a>] do_page_fault+0x1a/0x70 [ 2744.193452] [<ffffffff8175fa88>] page_fault+0x28/0x30 [ 2744.196140] Code: c7 08 50 a7 81 31 c0 e8 1c e1 ff ff 4c 89 e7 e8 b1 66 01 00 48 c7 c7 40 60 c0 81 e8 05 63 01 00 48 89 5d a8 4d 8b a7 08 03 00 00 <4d> 8d af 08 03 00 00 49 81 ec 18 03 00 00 eb 57 48 8b 83 a8 02 [ 2771.854441] BUG: soft lockup - CPU#0 stuck for 22s! [php5:15940] 3. lxc, cgroup, OOM, linux-image-3.13.0-32-generic, linux-image-3.13.0-35-generic, vsphere 4. Linux version 3.13.0-35-generic (buildd@akateko) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #62~precise1-Ubuntu SMP Mon Aug 18 14:52:04 UTC 2014 7.1 stackato@stackato-h5jt:~$ ./ver_linux.sh ./ver_linux.sh: line 1: ng: command not found If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS" NAME="Ubuntu" VERSION="12.04.4 LTS, Precise Pangolin" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu precise (12.04.4 LTS)" VERSION_ID="12.04" Linux stackato-h5jt 3.13.0-35-generic #62~precise1-Ubuntu SMP Mon Aug 18 14:52:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Gnu C gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3 Gnu make 3.81 util-linux linux 2.20.1 mount linux 2.20.1 (with libblkid and selinux support) modutils 3.16 e2fsprogs 1.42 PPP 2.4.5 Linux C Library > libc.2.15 Dynamic linker (ldd) 2.15 Procps 3.2.8 Net-tools 1.60 iproute2 iproute2-ss111117 Kbd 1.15.2 Sh-utils 8.13 Modules Loaded xt_nat veth ipt_REJECT xt_tcpudp xt_state xt_addrtype xt_conntrack iptable_filter aufs ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc vmwgfx ttm psmouse vmw_vmci drm vmw_balloon shpchp mac_hid i2c_piix4 serio_raw ppdev parport_pc lp parport floppy mptspi mptscsih e1000 mptbase vmw_pvscsi vmxnet3 free reports: total used free shared buffers cached Mem: 1918540 786944 1131596 0 82684 278140 -/+ buffers/cache: 426120 1492420 Swap: 1951740 0 1951740 7.2 /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz stepping : 2 microcode : 0x13 cpu MHz : 2400.085 cache size : 12288 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida arat epb dtherm bogomips : 4800.17 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz stepping : 2 microcode : 0x13 cpu MHz : 2400.085 cache size : 12288 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida arat epb dtherm bogomips : 4800.17 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: 7.3 cat /proc/modules xt_nat 12726 26 - Live 0x0000000000000000 veth 13331 0 - Live 0x0000000000000000 ipt_REJECT 12576 0 - Live 0x0000000000000000 xt_tcpudp 12924 39 - Live 0x0000000000000000 xt_state 12578 13 - Live 0x0000000000000000 xt_addrtype 12713 2 - Live 0x0000000000000000 xt_conntrack 12760 1 - Live 0x0000000000000000 iptable_filter 12810 1 - Live 0x0000000000000000 aufs 210175 999 - Live 0x0000000000000000 ipt_MASQUERADE 12880 2 - Live 0x0000000000000000 iptable_nat 13151 1 - Live 0x0000000000000000 nf_conntrack_ipv4 15063 15 - Live 0x0000000000000000 nf_defrag_ipv4 12758 1 nf_conntrack_ipv4, Live 0x0000000000000000 nf_nat_ipv4 13316 1 iptable_nat, Live 0x0000000000000000 nf_nat 26091 4 xt_nat,ipt_MASQUERADE,iptable_nat,nf_nat_ipv4, Live 0x0000000000000000 nf_conntrack 97581 7 xt_state,xt_conntrack,ipt_MASQUERADE,iptable_nat,nf_conntrack_ipv4,nf_nat_ipv4,nf_nat, Live 0x0000000000000000 ip_tables 27716 2 iptable_filter,iptable_nat, Live 0x0000000000000000 x_tables 34194 9 xt_nat,ipt_REJECT,xt_tcpudp,xt_state,xt_addrtype,xt_conntrack,iptable_filter,ipt_MASQUERADE,ip_tables, Live 0x0000000000000000 bridge 116291 0 - Live 0x0000000000000000 stp 12976 1 bridge, Live 0x0000000000000000 llc 14597 2 bridge,stp, Live 0x0000000000000000 vmwgfx 185498 1 - Live 0x0000000000000000 ttm 90134 1 vmwgfx, Live 0x0000000000000000 psmouse 113295 0 - Live 0x0000000000000000 vmw_vmci 68525 0 - Live 0x0000000000000000 drm 308868 2 vmwgfx,ttm, Live 0x0000000000000000 vmw_balloon 13593 0 - Live 0x0000000000000000 shpchp 37201 0 - Live 0x0000000000000000 mac_hid 13253 0 - Live 0x0000000000000000 i2c_piix4 22299 0 - Live 0x0000000000000000 serio_raw 13462 0 - Live 0x0000000000000000 ppdev 17711 0 - Live 0x0000000000000000 parport_pc 32866 1 - Live 0x0000000000000000 lp 17799 0 - Live 0x0000000000000000 parport 42481 3 ppdev,parport_pc,lp, Live 0x0000000000000000 floppy 70207 0 - Live 0x0000000000000000 mptspi 22921 3 - Live 0x0000000000000000 mptscsih 44751 1 mptspi, Live 0x0000000000000000 e1000 152011 0 - Live 0x0000000000000000 mptbase 103162 2 mptspi,mptscsih, Live 0x0000000000000000 vmw_pvscsi 23372 0 - Live 0x0000000000000000 vmxnet3 50657 0 - Live 0x0000000000000000 Thanks, Tyler -- To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
