Sorry for late reply. Seems we stuck in the while loop in mem_cgroup_reparent_charges(). I talked with Michal during Kernel Summit, and seems Google also hit this bug. Let's get more people involed. On 2013/10/18 17:57, Markus Blank-Burian wrote: > My test-runs now reproduced the bug with tracing enabled. The mutex > holding thread is definitely the one I posted earlier, and with the > "-t" option the crash utility can also display the whole stack > backtrace. (Did only show the first 3 lines without this options, > which confused me earlier into thinking, that the worker thread was > idle). I will keep the test machine running in this state if you need > more information. > > crash> bt 13115 -t > PID: 13115 TASK: ffff88082e34a050 CPU: 4 COMMAND: "kworker/4:0" > START: __schedule at ffffffff813e0f4f > [ffff88082f673ad8] schedule at ffffffff813e111f > [ffff88082f673ae8] schedule_timeout at ffffffff813ddd6c > [ffff88082f673af8] mark_held_locks at ffffffff8107bec4 > [ffff88082f673b10] _raw_spin_unlock_irq at ffffffff813e2625 > [ffff88082f673b38] trace_hardirqs_on_caller at ffffffff8107c04f > [ffff88082f673b58] trace_hardirqs_on at ffffffff8107c078 > [ffff88082f673b80] __wait_for_common at ffffffff813e0980 > [ffff88082f673b88] schedule_timeout at ffffffff813ddd38 > [ffff88082f673ba0] default_wake_function at ffffffff8105a258 > [ffff88082f673bb8] call_rcu at ffffffff810a552b > [ffff88082f673be8] wait_for_completion at ffffffff813e0a1c > [ffff88082f673bf8] wait_rcu_gp at ffffffff8104c736 > [ffff88082f673c08] wakeme_after_rcu at ffffffff8104c6d1 > [ffff88082f673c60] __mutex_unlock_slowpath at ffffffff813e0217 > [ffff88082f673c88] synchronize_rcu at ffffffff810a3f50 > [ffff88082f673c98] mem_cgroup_reparent_charges at ffffffff810f6765 > [ffff88082f673d28] mem_cgroup_css_offline at ffffffff810f6b9f > [ffff88082f673d58] offline_css at ffffffff8108b4aa > [ffff88082f673d80] cgroup_offline_fn at ffffffff8108e112 > [ffff88082f673dc0] process_one_work at ffffffff810493b3 > [ffff88082f673dc8] process_one_work at ffffffff81049348 > [ffff88082f673e28] worker_thread at ffffffff81049d7b > [ffff88082f673e48] worker_thread at ffffffff81049c37 > [ffff88082f673e60] kthread at ffffffff8104ef80 > [ffff88082f673f28] kthread at ffffffff8104eed4 > [ffff88082f673f50] ret_from_fork at ffffffff813e31ec > [ffff88082f673f80] kthread at ffffffff8104eed4 > > On Fri, Oct 18, 2013 at 11:34 AM, Markus Blank-Burian > <burian@xxxxxxxxxxx> wrote: >> I guess I found out, where it is hanging: While waiting for the >> test-runs to trigger the bug, I tried "echo w > /proc/sysrq-trigger" >> to show the stacks of all blocked tasks, and one of them was always >> this one: >> >> [586147.824671] kworker/3:5 D ffff8800df81e208 0 10909 2 0x00000000 >> [586147.824671] Workqueue: events cgroup_offline_fn >> [586147.824671] ffff8800fba7bbd0 0000000000000002 ffff88007afc2ee0 >> ffff8800fba7bfd8 >> [586147.824671] ffff8800fba7bfd8 0000000000011c40 ffff8800df81ddc0 >> 7fffffffffffffff >> [586147.824671] ffff8800fba7bcf8 ffff8800df81ddc0 0000000000000002 >> ffff8800fba7bcf0 >> [586147.824671] Call Trace: >> [586147.824671] [<ffffffff813c57e4>] schedule+0x60/0x62 >> [586147.824671] [<ffffffff813c374c>] schedule_timeout+0x34/0x11c >> [586147.824671] [<ffffffff81053305>] ? __wake_up_common+0x51/0x7e >> [586147.824671] [<ffffffff813c6a73>] ? _raw_spin_unlock_irqrestore+0x29/0x34 >> [586147.824671] [<ffffffff813c5097>] __wait_for_common+0x9c/0x119 >> [586147.824671] [<ffffffff813c3718>] ? svcauth_gss_legacy_init+0x176/0x176 >> [586147.824671] [<ffffffff8105790d>] ? wake_up_state+0xd/0xd >> [586147.824671] [<ffffffff8109c237>] ? call_rcu_bh+0x18/0x18 >> [586147.824671] [<ffffffff813c5133>] wait_for_completion+0x1f/0x21 >> [586147.824671] [<ffffffff8104a8ee>] wait_rcu_gp+0x46/0x4c >> [586147.824671] [<ffffffff8104a899>] ? __rcu_read_unlock+0x4c/0x4c >> [586147.824671] [<ffffffff8109ad6b>] synchronize_rcu+0x29/0x2b >> [586147.824671] [<ffffffff810ec34e>] mem_cgroup_reparent_charges+0x63/0x2fb >> [586147.824671] [<ffffffff810ec75a>] mem_cgroup_css_offline+0xa5/0x14a >> [586147.824671] [<ffffffff8108329e>] offline_css.part.15+0x1b/0x2e >> [586147.824671] [<ffffffff81084f8b>] cgroup_offline_fn+0x72/0x137 >> [586147.824671] [<ffffffff81047cb7>] process_one_work+0x15f/0x21e >> [586147.824671] [<ffffffff81048159>] worker_thread+0x144/0x1f0 >> [586147.824671] [<ffffffff81048015>] ? rescuer_thread+0x275/0x275 >> [586147.824671] [<ffffffff8104cbec>] kthread+0x88/0x90 >> [586147.824671] [<ffffffff8104cb64>] ? __kthread_parkme+0x60/0x60 >> [586147.824671] [<ffffffff813c756c>] ret_from_fork+0x7c/0xb0 >> [586147.824671] [<ffffffff8104cb64>] ? __kthread_parkme+0x60/0x60 >> >> >> On Tue, Oct 15, 2013 at 5:15 AM, Li Zefan <lizefan@xxxxxxxxxx> wrote: >>> On 2013/10/14 16:06, Markus Blank-Burian wrote: >>>> The crash utility indicated, that the lock was held by a kworker >>>> thread, which was idle at the moment. So there might be a case, where >>>> no unlock is done. I am trying to reproduce the problem at the moment >>>> with CONFIG_PROVE_LOCKING, but without luck so far. It seems, that my >>>> test-job is quite bad at reproducing the bug. I'll let you know, if I >>>> can find out more. >>>> >>> >>> Thanks. I'll review the code to see if I can find some suspect. >>> >>> PS: I'll be travelling from 10/16 ~ 10/28, so I may not be able >>> to spend much time on this. -- To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
