H all, i did a dlm_tool leave clvmd on one node (node06) of a CMAN cluster with CLVMD Now I have the problem that clvmd is stuck and all nodes lost connections to DLM. For some reason dlm want's to fence member 8 I guess and that might stuck the whole dlm? All other stacks, cman, corosync look fine... Thanks, Bjoern Error: dlm: closing connection to node 2 dlm: closing connection to node 3 dlm: closing connection to node 4 dlm: closing connection to node 5 dlm: closing connection to node 6 dlm: closing connection to node 8 dlm: closing connection to node 9 dlm: closing connection to node 10 dlm: closing connection to node 2 dlm: closing connection to node 3 dlm: closing connection to node 4 dlm: closing connection to node 5 dlm: closing connection to node 6 dlm: closing connection to node 8 dlm: closing connection to node 9 dlm: closing connection to node 10 INFO: task dlm_tool:33699 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. dlm_tool D 0000000000000003 0 33699 33698 0x00000080 ffff88138905dcc0 0000000000000082 ffffffff81168043 ffff88138905dd18 ffff88138905dd08 ffff88305b30ccc0 ffff88304fa5c800 ffff883058e49900 ffff881857329058 ffff88138905dfd8 000000000000fb88 ffff881857329058 Call Trace: [<ffffffff81168043>] ? kmem_cache_alloc_trace+0x1a3/0x1b0 [<ffffffff8132f79a>] ? misc_open+0x1ca/0x320 [<ffffffff81510725>] rwsem_down_failed_common+0x95/0x1d0 [<ffffffff81185505>] ? chrdev_open+0x125/0x230 [<ffffffff815108b6>] rwsem_down_read_failed+0x26/0x30 [<ffffffff8117e5ff>] ? __dentry_open+0x23f/0x360 [<ffffffff81283894>] call_rwsem_down_read_failed+0x14/0x30 [<ffffffff8150fdb4>] ? down_read+0x24/0x30 [<ffffffffa06d948d>] dlm_clear_proc_locks+0x3d/0x2a0 [dlm] [<ffffffff811dfed6>] ? generic_acl_chmod+0x46/0xd0 [<ffffffffa06e4b36>] device_close+0x66/0xc0 [dlm] [<ffffffff81182b45>] __fput+0xf5/0x210 [<ffffffff81182c85>] fput+0x25/0x30 [<ffffffff8117e0dd>] filp_close+0x5d/0x90 [<ffffffff8117e1b5>] sys_close+0xa5/0x100 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b Status: cman_tool nodes Node Sts Inc Joined Name 1 M 18908 2014-03-24 19:01:00 node01 2 M 18972 2014-04-06 22:47:57 node02 3 M 18972 2014-04-06 22:47:57 node03 4 M 18972 2014-04-06 22:47:57 node04 5 M 18972 2014-04-06 22:47:57 node05 6 X 18960 node06 7 X 18928 node07 8 M 18972 2014-04-06 22:47:57 node08 9 M 18972 2014-04-06 22:47:57 node09 10 M 18972 2014-04-06 22:47:57 node10 dlm lockspaces name clvmd id 0x4104eefa flags 0x00000004 kern_stop change member 8 joined 0 remove 1 failed 0 seq 11,11 members 1 2 3 4 5 8 9 10 new change member 8 joined 1 remove 0 failed 0 seq 12,41 new status wait_messages 0 wait_condition 1 fencing new members 1 2 3 4 5 8 9 10 DLM dump: 1396849677 cluster node 2 added seq 18972 1396849677 set_configfs_node 2 10.14.18.66 local 0 1396849677 cluster node 3 added seq 18972 1396849677 set_configfs_node 3 10.14.18.67 local 0 1396849677 cluster node 4 added seq 18972 1396849677 set_configfs_node 4 10.14.18.68 local 0 1396849677 cluster node 5 added seq 18972 1396849677 set_configfs_node 5 10.14.18.70 local 0 1396849677 cluster node 8 added seq 18972 1396849677 set_configfs_node 8 10.14.18.80 local 0 1396849677 cluster node 9 added seq 18972 1396849677 set_configfs_node 9 10.14.18.81 local 0 1396849677 cluster node 10 added seq 18972 1396849677 set_configfs_node 10 10.14.18.77 local 0 1396849677 dlm:ls:clvmd conf 2 1 0 memb 1 3 join 3 left 1396849677 clvmd add_change cg 35 joined nodeid 3 1396849677 clvmd add_change cg 35 counts member 2 joined 1 remove 0 failed 0 1396849677 dlm:ls:clvmd conf 3 1 0 memb 1 2 3 join 2 left 1396849677 clvmd add_change cg 36 joined nodeid 2 1396849677 clvmd add_change cg 36 counts member 3 joined 1 remove 0 failed 0 1396849677 dlm:ls:clvmd conf 4 1 0 memb 1 2 3 9 join 9 left 1396849677 clvmd add_change cg 37 joined nodeid 9 1396849677 clvmd add_change cg 37 counts member 4 joined 1 remove 0 failed 0 1396849677 dlm:ls:clvmd conf 5 1 0 memb 1 2 3 8 9 join 8 left 1396849677 clvmd add_change cg 38 joined nodeid 8 1396849677 clvmd add_change cg 38 counts member 5 joined 1 remove 0 failed 0 1396849677 dlm:ls:clvmd conf 6 1 0 memb 1 2 3 8 9 10 join 10 left 1396849677 clvmd add_change cg 39 joined nodeid 10 1396849677 clvmd add_change cg 39 counts member 6 joined 1 remove 0 failed 0 1396849677 dlm:ls:clvmd conf 7 1 0 memb 1 2 3 5 8 9 10 join 5 left 1396849677 clvmd add_change cg 40 joined nodeid 5 1396849677 clvmd add_change cg 40 counts member 7 joined 1 remove 0 failed 0 1396849677 dlm:ls:clvmd conf 8 1 0 memb 1 2 3 4 5 8 9 10 join 4 left 1396849677 clvmd add_change cg 41 joined nodeid 4 1396849677 clvmd add_change cg 41 counts member 8 joined 1 remove 0 failed 0 1396849677 dlm:controld conf 2 1 0 memb 1 3 join 3 left 1396849677 dlm:controld conf 3 1 0 memb 1 2 3 join 2 left 1396849677 dlm:controld conf 4 1 0 memb 1 2 3 9 join 9 left 1396849677 dlm:controld conf 5 1 0 memb 1 2 3 8 9 join 8 left 1396849677 dlm:controld conf 6 1 0 memb 1 2 3 8 9 10 join 10 left 1396849677 dlm:controld conf 7 1 0 memb 1 2 3 5 8 9 10 join 5 left 1396849677 dlm:controld conf 8 1 0 memb 1 2 3 4 5 8 9 10 join 4 left -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
