On Mar 17, 2006, at 11:40 PM, David Teigland wrote:
On Fri, Mar 17, 2006 at 03:38:59PM -0600, Paul Tader wrote:Mar 17 11:38:02 nodename kernel: d0 unlock fe20017a no idGFS is trying to unlock a lock that doesn't exist which causes the panic. We know this happens if cman shuts down the dlm while it's in use (cmandoes this if it's lost connection with the cluster.) There's some new output in the RHEL4U3 dlm that should tell us if that's in fact what's happening or if there's some other cause that we need to uncover. So, you should look on all nodes for any cman messages in /var/log/messages or the console. And when you're using the latestversion look for the new dlm message "WARNING: dlm_emergency_shutdown".
Dave, We had similar problems on our 4 node GFS cluster: * STABLE from cvs * 2.6.16-rc5-sara3 I have attached the kern.log's. 1 cman and 3 dlm kernel panics. Regards
Mar 16 09:34:40 ifs1 kernel: kernel BUG at /usr/src/gfs/stable_1.0.2/stable/cluster/cman-kernel/src/membership.c:3151! Mar 16 09:34:40 ifs1 kernel: invalid opcode: 0000 [#1] Mar 16 09:34:40 ifs1 kernel: SMP Mar 16 09:34:40 ifs1 kernel: Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod Mar 16 09:34:40 ifs1 kernel: CPU: 1 Mar 16 09:34:40 ifs1 kernel: EIP: 0060:[<f8ac7825>] Tainted: GF VLI Mar 16 09:34:40 ifs1 kernel: EFLAGS: 00010246 (2.6.16-rc5-sara3 #1) Mar 16 09:34:40 ifs1 kernel: EIP is at elect_master+0x34/0x41 [cman] Mar 16 09:34:40 ifs1 kernel: eax: f8a3e000 ebx: 00000080 ecx: 00000080 edx: 00000000 Mar 16 09:34:40 ifs1 kernel: esi: f8adb584 edi: f655ffcc ebp: f655ff7c esp: f655ff78 Mar 16 09:34:40 ifs1 kernel: ds: 007b es: 007b ss: 0068 Mar 16 09:34:40 ifs1 kernel: Process cman_memb (pid: 5643, threadinfo=f655e000 task=f7de5550) Mar 16 09:34:40 ifs1 kernel: Stack: <0>f5cd1800 f655ff90 f8ac5430 f655ff98 00000000 f5cd1800 f655ffa4 f8ac3acc Mar 16 09:34:40 ifs1 kernel: f6dfe840 00000000 f7de5550 f655ffe4 f8ac3cb3 0000001f 00000000 c0102612 Mar 16 09:34:40 ifs1 kernel: 00000000 f7de5550 c01124e1 00100100 00200200 00000000 00000000 00000000 Mar 16 09:34:40 ifs1 kernel: Call Trace: Mar 16 09:34:40 ifs1 kernel: [<c0103599>] show_stack_log_lvl+0xad/0xb5 Mar 16 09:34:40 ifs1 kernel: [<c01036db>] show_registers+0x10d/0x176 Mar 16 09:34:40 ifs1 kernel: [<c01038ad>] die+0xf2/0x16d Mar 16 09:34:40 ifs1 kernel: [<c0103996>] do_trap+0x6e/0x8a Mar 16 09:34:40 ifs1 kernel: [<c0103bed>] do_invalid_op+0x90/0x97 Mar 16 09:34:40 ifs1 kernel: [<c010322f>] error_code+0x4f/0x54 Mar 16 09:34:40 ifs1 kernel: [<f8ac5430>] a_node_just_died+0x118/0x178 [cman] Mar 16 09:34:40 ifs1 kernel: [<f8ac3acc>] process_dead_nodes+0x4e/0x7a [cman] Mar 16 09:34:40 ifs1 kernel: [<f8ac3cb3>] membership_kthread+0x1bb/0x38d [cman] Mar 16 09:34:40 ifs1 kernel: [<c0100ed9>] kernel_thread_helper+0x5/0xb Mar 16 09:34:40 ifs1 kernel: Code: 8b 1d 44 c3 ad f8 39 d9 7d 21 a1 48 c3 ad f8 8b 14 88 85 d2 74 10 83 7a 1c 02 75 0a 8b 45 08 89 10 8b 42 14 eb 0f 41 39 d9 7c df <0f> 0b 4f 0c 60 f2 ac f8 31 c0 5b 5d c3 55 89 e5 ff 35 48 c3 ad
Attachment:
dlm_kernel_crash.txt3.gz
Description: GNU Zip compressed data
Mar 16 09:34:34 ifs3 kernel: WARNING: dlm_emergency_shutdown Mar 16 09:34:34 ifs3 kernel: WARNING: dlm_emergency_shutdown Mar 16 09:34:34 ifs3 kernel: SM: 00000001 sm_stop: SG still joined Mar 16 09:34:34 ifs3 kernel: SM: 01000002 sm_stop: SG still joined Mar 16 09:34:34 ifs3 kernel: SM: 02000004 sm_stop: SG still joined Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg1_lv1 send einval to 2 Mar 16 09:34:35 ifs3 kernel: lisa_vg4_lv2 (6204) req reply einval 8690181 fr 2 r 2 1 Mar 16 09:34:35 ifs3 kernel: pr_start count 4 type 2 event 60 flags 21a Mar 16 09:34:35 ifs3 kernel: 10879 pr_start 60 done 1 Mar 16 09:34:35 ifs3 kernel: 10879 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10899 pr_start last_stop 37 last_start 62 last_finish 37 Mar 16 09:34:35 ifs3 kernel: 10899 pr_start count 4 type 2 event 62 flags 21a Mar 16 09:34:35 ifs3 kernel: 10899 pr_start 62 done 1 Mar 16 09:34:35 ifs3 kernel: 10899 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10918 pr_start last_stop 40 last_start 64 last_finish 40 Mar 16 09:34:35 ifs3 kernel: 10918 pr_start count 4 type 2 event 64 flags 21a Mar 16 09:34:35 ifs3 kernel: 10918 pr_start 64 done 1 Mar 16 09:34:35 ifs3 kernel: 10918 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10936 pr_start last_stop 42 last_start 66 last_finish 42 Mar 16 09:34:35 ifs3 kernel: 10936 pr_start count 4 type 2 event 66 flags 21a Mar 16 09:34:35 ifs3 kernel: 10936 pr_start 66 done 1 Mar 16 09:34:35 ifs3 kernel: 10936 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 11203 pr_start last_stop 44 last_start 68 last_finish 44 Mar 16 09:34:35 ifs3 kernel: 11203 pr_start count 4 type 2 event 68 flags 21a Mar 16 09:34:35 ifs3 kernel: 11203 pr_start 68 done 1 Mar 16 09:34:35 ifs3 kernel: 11203 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 11269 pr_start last_stop 46 last_start 70 last_finish 46 Mar 16 09:34:35 ifs3 kernel: 11269 pr_start count 4 type 2 event 70 flags 21a Mar 16 09:34:35 ifs3 kernel: 11269 pr_start 70 done 1 Mar 16 09:34:35 ifs3 kernel: 11269 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 11269 pr_start last_stop 70 last_start 71 last_finish 70 Mar 16 09:34:35 ifs3 kernel: 11269 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 11269 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 11269 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 11203 pr_start last_stop 68 last_start 71 last_finish 68 Mar 16 09:34:35 ifs3 kernel: 11203 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 11203 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 11203 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 10936 pr_start last_stop 66 last_start 71 last_finish 66 Mar 16 09:34:35 ifs3 kernel: 10936 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10936 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10936 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 10918 pr_start last_stop 64 last_start 71 last_finish 64 Mar 16 09:34:35 ifs3 kernel: 10918 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10918 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10918 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 10899 pr_start last_stop 62 last_start 71 last_finish 62 Mar 16 09:34:35 ifs3 kernel: 10899 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10899 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10899 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 11207 recovery_done jid 3 msg 308 11a Mar 16 09:34:35 ifs3 kernel: 11207 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 11207 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10879 pr_start last_stop 60 last_start 71 last_finish 60 Mar 16 09:34:35 ifs3 kernel: 10879 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10879 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10879 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 11171 recovery_done jid 3 msg 308 11a Mar 16 09:34:35 ifs3 kernel: 11171 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 11171 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10860 pr_start last_stop 58 last_start 71 last_finish 58 Mar 16 09:34:35 ifs3 kernel: 10860 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10860 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10860 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 10841 pr_start last_stop 56 last_start 71 last_finish 56 Mar 16 09:34:35 ifs3 kernel: 10841 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10841 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10841 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 10902 recovery_done jid 3 msg 308 11a Mar 16 09:34:35 ifs3 kernel: 10902 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 10902 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10822 pr_start last_stop 54 last_start 71 last_finish 54 Mar 16 09:34:35 ifs3 kernel: 10822 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10822 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10822 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 10921 recovery_done jid 3 msg 308 11a Mar 16 09:34:35 ifs3 kernel: 10921 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 10921 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 11288 recovery_done jid 3 msg 308 11a Mar 16 09:34:35 ifs3 kernel: 11288 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 11288 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10803 pr_start last_stop 52 last_start 71 last_finish 52 Mar 16 09:34:35 ifs3 kernel: 10803 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10803 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10803 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 10864 recovery_done jid 3 msg 308 11a Mar 16 09:34:35 ifs3 kernel: 10864 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 10864 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10784 pr_start last_stop 50 last_start 71 last_finish 50 Mar 16 09:34:35 ifs3 kernel: 10784 pr_start count 3 type 1 event 71 flags 21a Mar 16 09:34:35 ifs3 kernel: 10784 pr_start cb jid 3 id 4 Mar 16 09:34:35 ifs3 kernel: 10784 pr_start 71 done 0 Mar 16 09:34:35 ifs3 kernel: 10826 recovery_done jid 3 msg 308 11a Mar 16 09:34:35 ifs3 kernel: 10826 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 10826 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10845 recovery_done jid 3 msg 308 11a Mar 16 09:34:35 ifs3 kernel: 10845 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 10845 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10861 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10842 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10823 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10883 recovery_done jid 3 msg 309 11a Mar 16 09:34:35 ifs3 kernel: 10883 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 10883 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10879 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 11203 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10788 recovery_done jid 3 msg 309 11a Mar 16 09:34:35 ifs3 kernel: 10788 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 10788 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10785 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10917 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10937 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 11269 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10807 recovery_done jid 3 msg 309 11a Mar 16 09:34:35 ifs3 kernel: 10807 recovery_done nodeid 4 flg 1b Mar 16 09:34:35 ifs3 kernel: 10807 recovery_done start_done 71 Mar 16 09:34:35 ifs3 kernel: 10803 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: 10898 pr_finish flags 1a Mar 16 09:34:35 ifs3 kernel: Mar 16 09:34:35 ifs3 kernel: lock_dlm: Assertion failed on line 428 of file /usr/src/gfs/stable_1.0.2/stable/cluster/gfs-kernel/src/dlm/lock.c Mar 16 09:34:35 ifs3 kernel: lock_dlm: assertion: "!error" Mar 16 09:34:35 ifs3 kernel: lock_dlm: time = 149952310 Mar 16 09:34:35 ifs3 kernel: lisa_vg4_lv1: num=2,1a err=-22 cur=-1 req=3 lkf=10000 Mar 16 09:34:35 ifs3 kernel: Mar 16 09:34:35 ifs3 kernel: ------------[ cut here ]------------ Mar 16 09:34:35 ifs3 kernel: kernel BUG at /usr/src/gfs/stable_1.0.2/stable/cluster/gfs-kernel/src/dlm/lock.c:428! Mar 16 09:34:35 ifs3 kernel: invalid opcode: 0000 [#1] Mar 16 09:34:35 ifs3 kernel: SMP Mar 16 09:34:35 ifs3 kernel: Modules linked in: lock_dlm dlm cman dm_round_robin dm_multipath sg ide_floppy ide_cd cdrom qla2xxx siimage piix e1000 gfs lock_harness dm_mod Mar 16 09:34:35 ifs3 kernel: CPU: 0 Mar 16 09:34:35 ifs3 kernel: EIP: 0060:[<f8aa5714>] Tainted: GF VLI Mar 16 09:34:35 ifs3 kernel: EFLAGS: 00010246 (2.6.16-rc5-sara3 #1) Mar 16 09:34:35 ifs3 kernel: EIP is at do_dlm_lock+0x138/0x152 [lock_dlm] Mar 16 09:34:35 ifs3 kernel: eax: 00000004 ebx: ffffffea ecx: 0000c834 edx: 00000246 Mar 16 09:34:35 ifs3 kernel: esi: f5b832c0 edi: f5ee62c0 ebp: f65afd44 esp: f65afd10 Mar 16 09:34:35 ifs3 kernel: ds: 007b es: 007b ss: 0068 Mar 16 09:34:35 ifs3 kernel: Process exportfs (pid: 14177, threadinfo=f65ae000 task=f7f00a70) Mar 16 09:34:35 ifs3 kernel: Stack: <0>f8aa9d89 00000001 20202020 32202020 20202020 20202020 20202020 61312020 Mar 16 09:34:35 ifs3 kernel: c0150018 f4276294 f5b832c0 00000008 f8bcf000 f65afd78 f8aa57b9 f5b832c0 Mar 16 09:34:35 ifs3 kernel: 00000000 f5b832c0 00000008 ffffffff 00000003 00000003 f5b832c0 00000000 Mar 16 09:34:35 ifs3 kernel: Call Trace: Mar 16 09:34:35 ifs3 kernel: [<c0103599>] show_stack_log_lvl+0xad/0xb5 Mar 16 09:34:35 ifs3 kernel: [<c01036db>] show_registers+0x10d/0x176 Mar 16 09:34:35 ifs3 kernel: [<c01038ad>] die+0xf2/0x16d Mar 16 09:34:35 ifs3 kernel: [<c0103996>] do_trap+0x6e/0x8a Mar 16 09:34:35 ifs3 kernel: [<c0103bed>] do_invalid_op+0x90/0x97 Mar 16 09:34:35 ifs3 kernel: [<c010322f>] error_code+0x4f/0x54 Mar 16 09:34:35 ifs3 kernel: [<f8aa57b9>] lm_dlm_lock+0x4f/0x5b [lock_dlm] Mar 16 09:34:35 ifs3 kernel: [<f899a776>] gfs_lm_lock+0x32/0x4c [gfs] Mar 16 09:34:35 ifs3 kernel: [<f89909f8>] gfs_glock_xmote_th+0x125/0x161 [gfs] Mar 16 09:34:35 ifs3 kernel: [<f8993625>] inode_go_xmote_th+0x20/0x25 [gfs] Mar 16 09:34:35 ifs3 kernel: [<f89900fd>] rq_promote+0xb3/0x136 [gfs] Mar 16 09:34:35 ifs3 kernel: [<f89902e3>] run_queue+0x85/0xbb [gfs] Mar 16 09:34:35 ifs3 kernel: [<f8991281>] gfs_glock_nq+0xce/0x119 [gfs] Mar 16 09:34:35 ifs3 kernel: [<f89917f0>] gfs_glock_nq_init+0x1d/0x36 [gfs] Mar 16 09:34:35 ifs3 kernel: [<f89a6963>] gfs_getattr+0x35/0x5a [gfs] Mar 16 09:34:35 ifs3 kernel: [<c01535d2>] vfs_getattr+0x23/0x78 Mar 16 09:34:35 ifs3 kernel: [<c01536ac>] vfs_lstat_fd+0x2e/0x42 Mar 16 09:34:35 ifs3 kernel: [<c01536d0>] vfs_lstat+0x10/0x12 Mar 16 09:34:35 ifs3 kernel: [<c0153c35>] sys_lstat64+0x13/0x29 Mar 16 09:34:35 ifs3 kernel: [<c0102709>] syscall_call+0x7/0xb Mar 16 09:34:35 ifs3 kernel: Code: 26 50 0f bf 46 24 50 53 ff 76 08 ff 76 04 ff 76 0c ff 77 18 68 e0 a6 aa f8 e8 f2 17 67 c7 83 c4 38 68 89 9d aa f8 e8 e5 17 67 c7 <0f> 0b ac 01 c0 a4 aa f8 68 a0 a5 aa f8 e8 99 10 67 c7 8d 65 f4
Attachment:
dlm_lock_crash.txt2
Description: Binary data
-- Bas van der Vlies basv@xxxxxxx
-- Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
