On 08/19/2012 05:52 AM, Bart Verwilst wrote: > Hi, > > I have a 3-node cluster in testing which seem to work quite well ( cman, > rgmanager, gfs2, etc ). > On (only) one of my nodes, yesterday i noticed the message below in dmesg. > > I saw this 30 minutes after the facts. I could browse both my gfs2 > mounts, there was no fencing or anything on any node. > > Any idea what might have caused this, and then go away? > > Aug 19 00:10:01 vm02-test kernel: [282120.240067] INFO: task > kworker/1:0:3117 blocked for more than 120 seconds. > Aug 19 00:10:01 vm02-test kernel: [282120.240182] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Aug 19 00:10:01 vm02-test kernel: [282120.240296] kworker/1:0 D > ffff88032fc93900 0 3117 2 0x00000000 > Aug 19 00:10:01 vm02-test kernel: [282120.240302] ffff8802bb4dfb30 > 0000000000000046 ffff88031e4744d0 ffff8802bb4dffd8 > Aug 19 00:10:01 vm02-test kernel: [282120.240307] ffff8802bb4dffd8 > ffff8802bb4dffd8 ffff88031e5796f0 ffff88031e4744d0 > Aug 19 00:10:01 vm02-test kernel: [282120.240311] 0000000000000286 > ffff88032ffbd0f8 ffff8802bb4dfbc8 0000000000000002 > Aug 19 00:10:01 vm02-test kernel: [282120.240316] Call Trace: > Aug 19 00:10:01 vm02-test kernel: [282120.240334] [<ffffffffa0570290>] > ? gfs2_glock_demote_wait+0x20/0x20 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240340] [<ffffffff81666c89>] > schedule+0x29/0x70 > Aug 19 00:10:01 vm02-test kernel: [282120.240349] [<ffffffffa057029e>] > gfs2_glock_holder_wait+0xe/0x20 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240352] [<ffffffff81665400>] > __wait_on_bit+0x60/0x90 > Aug 19 00:10:01 vm02-test kernel: [282120.240361] [<ffffffffa0570290>] > ? gfs2_glock_demote_wait+0x20/0x20 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240364] [<ffffffff816654ac>] > out_of_line_wait_on_bit+0x7c/0x90 > Aug 19 00:10:01 vm02-test kernel: [282120.240369] [<ffffffff81073400>] > ? autoremove_wake_function+0x40/0x40 > Aug 19 00:10:01 vm02-test kernel: [282120.240378] [<ffffffffa05713a7>] > wait_on_holder+0x47/0x80 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240388] [<ffffffffa05741d8>] > gfs2_glock_nq+0x328/0x450 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240399] [<ffffffffa058a8ca>] > gfs2_check_blk_type+0x4a/0x150 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240410] [<ffffffffa058a8c1>] > ? gfs2_check_blk_type+0x41/0x150 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240421] [<ffffffffa058ba0c>] > gfs2_evict_inode+0x2cc/0x360 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240432] [<ffffffffa058b842>] > ? gfs2_evict_inode+0x102/0x360 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240437] [<ffffffff811940c2>] > evict+0xb2/0x1b0 > Aug 19 00:10:01 vm02-test kernel: [282120.240440] [<ffffffff811942c9>] > iput+0x109/0x210 > Aug 19 00:10:01 vm02-test kernel: [282120.240448] [<ffffffffa0572fdc>] > delete_work_func+0x5c/0x90 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240453] [<ffffffff8106d5fa>] > process_one_work+0x12a/0x420 > Aug 19 00:10:01 vm02-test kernel: [282120.240462] [<ffffffffa0572f80>] > ? gfs2_holder_uninit+0x40/0x40 [gfs2] > Aug 19 00:10:01 vm02-test kernel: [282120.240465] [<ffffffff8106e19e>] > worker_thread+0x12e/0x2f0 > Aug 19 00:10:01 vm02-test kernel: [282120.240469] [<ffffffff8106e070>] > ? manage_workers.isra.25+0x200/0x200 > Aug 19 00:10:01 vm02-test kernel: [282120.240472] [<ffffffff81072e73>] > kthread+0x93/0xa0 > Aug 19 00:10:01 vm02-test kernel: [282120.240477] [<ffffffff816710a4>] > kernel_thread_helper+0x4/0x10 > Aug 19 00:10:01 vm02-test kernel: [282120.240480] [<ffffffff81072de0>] > ? flush_kthread_worker+0x80/0x80 > Aug 19 00:10:01 vm02-test kernel: [282120.240484] [<ffffffff816710a0>] > ? gs_change+0x13/0x13 > Aug 19 00:12:01 vm02-test kernel: [282240.240061] INFO: task > kworker/1:0:3117 blocked for more than 120 seconds. > Aug 19 00:12:01 vm02-test kernel: [282240.240175] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Aug 19 00:12:01 vm02-test kernel: [282240.240289] kworker/1:0 D > ffff88032fc93900 0 3117 2 0x00000000 > Aug 19 00:12:01 vm02-test kernel: [282240.240294] ffff8802bb4dfb30 > 0000000000000046 ffff88031e4744d0 ffff8802bb4dffd8 > Aug 19 00:12:01 vm02-test kernel: [282240.240299] ffff8802bb4dffd8 > ffff8802bb4dffd8 ffff88031e5796f0 ffff88031e4744d0 > Aug 19 00:12:01 vm02-test kernel: [282240.240304] 0000000000000286 > ffff88032ffbd0f8 ffff8802bb4dfbc8 0000000000000002 > Aug 19 00:12:01 vm02-test kernel: [282240.240309] Call Trace: > Aug 19 00:12:01 vm02-test kernel: [282240.240326] [<ffffffffa0570290>] > ? gfs2_glock_demote_wait+0x20/0x20 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240332] [<ffffffff81666c89>] > schedule+0x29/0x70 > Aug 19 00:12:01 vm02-test kernel: [282240.240341] [<ffffffffa057029e>] > gfs2_glock_holder_wait+0xe/0x20 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240345] [<ffffffff81665400>] > __wait_on_bit+0x60/0x90 > Aug 19 00:12:01 vm02-test kernel: [282240.240353] [<ffffffffa0570290>] > ? gfs2_glock_demote_wait+0x20/0x20 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240357] [<ffffffff816654ac>] > out_of_line_wait_on_bit+0x7c/0x90 > Aug 19 00:12:01 vm02-test kernel: [282240.240362] [<ffffffff81073400>] > ? autoremove_wake_function+0x40/0x40 > Aug 19 00:12:01 vm02-test kernel: [282240.240371] [<ffffffffa05713a7>] > wait_on_holder+0x47/0x80 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240380] [<ffffffffa05741d8>] > gfs2_glock_nq+0x328/0x450 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240391] [<ffffffffa058a8ca>] > gfs2_check_blk_type+0x4a/0x150 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240402] [<ffffffffa058a8c1>] > ? gfs2_check_blk_type+0x41/0x150 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240413] [<ffffffffa058ba0c>] > gfs2_evict_inode+0x2cc/0x360 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240424] [<ffffffffa058b842>] > ? gfs2_evict_inode+0x102/0x360 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240429] [<ffffffff811940c2>] > evict+0xb2/0x1b0 > Aug 19 00:12:01 vm02-test kernel: [282240.240432] [<ffffffff811942c9>] > iput+0x109/0x210 > Aug 19 00:12:01 vm02-test kernel: [282240.240440] [<ffffffffa0572fdc>] > delete_work_func+0x5c/0x90 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240445] [<ffffffff8106d5fa>] > process_one_work+0x12a/0x420 > Aug 19 00:12:01 vm02-test kernel: [282240.240454] [<ffffffffa0572f80>] > ? gfs2_holder_uninit+0x40/0x40 [gfs2] > Aug 19 00:12:01 vm02-test kernel: [282240.240458] [<ffffffff8106e19e>] > worker_thread+0x12e/0x2f0 > Aug 19 00:12:01 vm02-test kernel: [282240.240462] [<ffffffff8106e070>] > ? manage_workers.isra.25+0x200/0x200 > Aug 19 00:12:01 vm02-test kernel: [282240.240465] [<ffffffff81072e73>] > kthread+0x93/0xa0 > Aug 19 00:12:01 vm02-test kernel: [282240.240469] [<ffffffff816710a4>] > kernel_thread_helper+0x4/0x10 > Aug 19 00:12:01 vm02-test kernel: [282240.240473] [<ffffffff81072de0>] > ? flush_kthread_worker+0x80/0x80 > Aug 19 00:12:01 vm02-test kernel: [282240.240476] [<ffffffff816710a0>] > ? gs_change+0x13/0x13 > <snip, goes on for a while> > > Kind regards, > > Bart I usually see this when DLM is blocked. DLM usually blocks on a failed fence action. To clarify; this comes up on one of the three nodes only? On the node with these messages, you shouldn't be able to look at the hung FS on the effected node. Can you share you versions and cluster.conf please? Also, what is in the logs in the three or four minutes before these messages start? Anything interesting in the log files of the other nodes around the same time period? digimer -- Digimer Papers and Projects: https://alteeve.com -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
