well,
today, I try to unmount GFS form one node to update it (for the latest
kernel).
All nodes had a kernel panic.
Here is the stack :
May 24 07:23:09 ancona kernel: CMAN: too many transition restarts - will die
May 24 07:23:09 ancona kernel: CMAN: we are leaving the cluster.
Inconsistent cluster view
May 24 07:23:09 ancona kernel: WARNING: dlm_emergency_shutdown
May 24 07:23:09 ancona clurgmgrd[4604]: <warning> #67: Shutting down
uncleanly
May 24 07:23:09 ancona kernel: WARNING: dlm_emergency_shutdown
May 24 07:23:09 ancona kernel: SM: 00000006 sm_stop: SG still joined
May 24 07:23:09 ancona kernel: SM: 01000008 sm_stop: SG still joined
May 24 07:23:09 ancona kernel: SM: 02000014 sm_stop: SG still joined
May 24 07:23:09 ancona kernel: SM: 0300000a sm_stop: SG still joined
May 24 07:23:09 ancona ccsd[3732]: Cluster manager shutdown. Attemping
to reconnect...
May 24 07:23:10 ancona kernel: dlm: dlm_unlock: lkid 947100ed lockspace
not found
May 24 07:23:10 ancona kernel: nval 91ed0131 fr 8 r 8 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 921a00a3 fr
8 r 8 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 924b0156 fr
7 r 7 2
May 24 07:23:10 ancona kernel: home send einval to 7
May 24 07:23:10 ancona kernel: home (3942) req reply einval 934f0161 fr
1 r 1 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 90a603ad fr
8 r 8 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 92b600d0 fr
4 r 4 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 915b02a7 fr
5 r 5 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 935b0262 fr
5 r 5 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 922d0261 fr
5 r 5 2
May 24 07:23:10 ancona kernel: home send einval to 2
May 24 07:23:10 ancona kernel: home send einval to 8
May 24 07:23:10 ancona kernel: home (3942) req reply einval 92b00008 fr
7 r 7 2
May 24 07:23:10 ancona kernel: home send einval to 7
May 24 07:23:10 ancona kernel: home (3942) req reply einval 92ca0337 fr
1 r 1 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 932d0128 fr
1 r 1 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 9276022a fr
7 r 7 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 94a90311 fr
8 r 8 2
May 24 07:23:10 ancona kernel: home (3942) req reply einval 93ec0156 fr
8 r 8 2
May 24 07:23:10 ancona kernel: 3931 pr_start last_stop 0 last_start 6
last_finish 0
May 24 07:23:10 ancona kernel: 3931 pr_start count 7 type 2 event 6
flags 250
May 24 07:23:10 ancona kernel: 3931 claim_jid 4
May 24 07:23:10 ancona kernel: 3931 pr_start 6 done 1
May 24 07:23:10 ancona kernel: 3931 pr_finish flags 5a
May 24 07:23:10 ancona kernel: 3916 recovery_done jid 4 msg 309 a
May 24 07:23:10 ancona kernel: 3916 recovery_done nodeid 6 flg 18
May 24 07:23:10 ancona kernel: 3930 pr_start last_stop 6 last_start 8
last_finish 6
May 24 07:23:10 ancona kernel: 3930 pr_start count 8 type 2 event 8
flags 21a
May 24 07:23:10 ancona kernel: 3930 pr_start 8 done 1
May 24 07:23:10 ancona kernel: 3930 pr_finish flags 1a
May 24 07:23:10 ancona kernel:
May 24 07:23:10 ancona kernel: lock_dlm: Assertion failed on line 361
of file /builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/dlm/lock.c
May 24 07:23:10 ancona kernel: lock_dlm: assertion: "!error || (plock
&& error == -EINPROGRESS)"
May 24 07:23:10 ancona kernel: lock_dlm: time = 2227212828
May 24 07:23:10 ancona kernel: home: error=-22 num=5,9641cab lkf=0 flags=84
May 24 07:23:10 ancona kernel:
May 24 07:23:10 ancona kernel: ------------[ cut here ]------------
May 24 07:23:10 ancona kernel: kernel BUG at
/builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/dlm/lock.c:361!
May 24 07:23:10 ancona kernel: invalid operand: 0000 [#1]
May 24 07:23:10 ancona kernel: SMP
May 24 07:23:11 ancona kernel: Modules linked in: autofs4 lock_dlm(U)
gfs(U) lock_harness(U) dlm(U) cman(U) md5 ipv6 sunrpc arpt_mangle
arptable_filter arp_tables dm_mirror dm_round_robin dm_multipath button
battery ac ohci_hcd tg3 bonding(U) floppy sg st ext3 jbd dm_mod qla2300
qla2xxx scsi_transport_fc cciss sd_mod scsi_mod
May 24 07:23:11 ancona kernel: CPU: 1
May 24 07:23:11 ancona kernel: EIP: 0060:[<f8ae8611>] Not tainted VLI
May 24 07:23:11 ancona kernel: EFLAGS: 00010246 (2.6.9-67.0.7.ELsmp)
May 24 07:23:11 ancona kernel: EIP is at do_dlm_unlock+0xa9/0xbf [lock_dlm]
May 24 07:23:11 ancona kernel: eax: 00000001 ebx: e82b5b80 ecx:
f6cdbef0 edx: f8aed2d3
May 24 07:23:11 ancona kernel: esi: ffffffea edi: 00000000 ebp:
f8a53000 esp: f6cdbeec
May 24 07:23:11 ancona kernel: ds: 007b es: 007b ss: 0068
May 24 07:23:11 ancona kernel: Process gfs_glockd (pid: 3939,
threadinfo=f6cdb000 task=f70d19b0)
May 24 07:23:11 ancona kernel: Stack: f8aed2d3 f8a53000 00000003
e82b5b80 f8ae88b2 f8b48ede 00000001 ea3e7874
May 24 07:23:11 ancona kernel: ea3e7858 f8b3ed63 f8b75e60
ed9fcac0 ea3e7858 f8b75e60 e6fcf8fc f8b3e257
May 24 07:23:11 ancona kernel: ea3e7858 00000001 ea3e7858
f8b3e30e ea3e7874 00000000 f8b3f5f2 00000000
May 24 07:23:11 ancona kernel: Call Trace:
May 24 07:23:11 ancona kernel: [<f8ae88b2>] lm_dlm_unlock+0x14/0x1c
[lock_dlm]
May 24 07:23:11 ancona kernel: [<f8b48ede>] gfs_lm_unlock+0x2c/0x42 [gfs]
May 24 07:23:11 ancona kernel: [<f8b3ed63>]
gfs_glock_drop_th+0xf3/0x12d [gfs]
May 24 07:23:11 ancona kernel: [<f8b3e257>] rq_demote+0x7f/0x98 [gfs]
May 24 07:23:11 ancona kernel: [<f8b3e30e>] run_queue+0x5a/0xc1 [gfs]
May 24 07:23:11 ancona kernel: [<f8b3f5f2>] gfs_glock_dq+0x15f/0x16e [gfs]
May 24 07:23:11 ancona kernel: [<f8b3f946>]
gfs_glock_dq_uninit+0x8/0x10 [gfs]
May 24 07:23:11 ancona kernel: [<f8b4251e>] gfs_inode_destroy+0x8e/0xbf
[gfs]
May 24 07:23:11 ancona kernel: [<f8b403c8>]
gfs_reclaim_glock+0xa2/0x13c [gfs]
May 24 07:23:11 ancona kernel: [<f8b32e05>] gfs_glockd+0x39/0xde [gfs]
May 24 07:23:11 ancona kernel: [<c011e7b9>] default_wake_function+0x0/0xc
May 24 07:23:11 ancona kernel: [<c02d8876>] ret_from_fork+0x6/0x14
May 24 07:23:11 ancona kernel: [<c011e7b9>] default_wake_function+0x0/0xc
May 24 07:23:11 ancona kernel: [<f8b32dcc>] gfs_glockd+0x0/0xde [gfs]
May 24 07:23:11 ancona kernel: [<c01041f5>] kernel_thread_helper+0x5/0xb
May 24 07:23:11 ancona kernel: Code: 73 34 8b 03 ff 73 2c ff 73 08 ff 73
04 ff 73 0c 56 ff 70 18 68 ef d3 ae f8 e8 de a2 63 c7 83 c4 34 68 d3 d2
ae f8 e8 d1 a2 63 c7 <0f> 0b 69 01 1b d2 ae f8 68 d5 d2 ae f8 e8 8c 9a
63 c7 5b 5e 5f
May 24 07:23:11 ancona kernel: <0>Fatal exception: panic in 5 seconds
May 24 07:23:11 ancona kernel: dlm: dlm_lock: no lockspace
May 24 07:23:12 ancona kernel: nval 91ed0131 fr 8 r 8 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 921a00a3 fr
8 r 8 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 924b0156 fr
7 r 7 2
May 24 07:23:12 ancona kernel: home send einval to 7
May 24 07:23:12 ancona kernel: home (3942) req reply einval 934f0161 fr
1 r 1 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 90a603ad fr
8 r 8 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 92b600d0 fr
4 r 4 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 915b02a7 fr
5 r 5 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 935b0262 fr
5 r 5 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 922d0261 fr
5 r 5 2
May 24 07:23:12 ancona kernel: home send einval to 2
May 24 07:23:12 ancona kernel: home send einval to 8
May 24 07:23:12 ancona kernel: home (3942) req reply einval 92b00008 fr
7 r 7 2
May 24 07:23:12 ancona kernel: home send einval to 7
May 24 07:23:12 ancona kernel: home (3942) req reply einval 92ca0337 fr
1 r 1 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 932d0128 fr
1 r 1 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 9276022a fr
7 r 7 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 94a90311 fr
8 r 8 2
May 24 07:23:12 ancona kernel: home (3942) req reply einval 93ec0156 fr
8 r 8 2
May 24 07:23:12 ancona kernel: 3931 pr_start last_stop 0 last_start 6
last_finish 0
May 24 07:23:12 ancona kernel: 3931 pr_start count 7 type 2 event 6
flags 250
May 24 07:23:12 ancona kernel: 3931 claim_jid 4
May 24 07:23:12 ancona kernel: 3931 pr_start 6 done 1
May 24 07:23:12 ancona kernel: 3931 pr_finish flags 5a
May 24 07:23:12 ancona kernel: 3916 recovery_done jid 4 msg 309 a
May 24 07:23:12 ancona kernel: 3916 recovery_done nodeid 6 flg 18
May 24 07:23:12 ancona kernel: 3930 pr_start last_stop 6 last_start 8
last_finish 6
May 24 07:23:12 ancona kernel: 3930 pr_start count 8 type 2 event 8
flags 21a
May 24 07:23:12 ancona kernel: 3930 pr_start 8 done 1
May 24 07:23:12 ancona kernel: 3930 pr_finish flags 1a
May 24 07:23:12 ancona kernel:
May 24 07:23:12 ancona kernel: lock_dlm: Assertion failed on line 432
of file /builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/dlm/lock.c
May 24 07:23:13 ancona kernel: lock_dlm: assertion: "!error"
May 24 07:23:13 ancona kernel: lock_dlm: time = 2227213341
May 24 07:23:13 ancona kernel: home: num=2,7a2ec26 err=-22 cur=-1 req=3
lkf=10000
May 24 07:23:13 ancona kernel:
May 24 07:23:13 ancona kernel: ------------[ cut here ]------------
May 24 07:23:13 ancona kernel: kernel BUG at
/builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/dlm/lock.c:432!
May 24 07:23:13 ancona kernel: invalid operand: 0000 [#2]
May 24 07:23:13 ancona kernel: SMP
May 24 07:23:13 ancona kernel: Modules linked in: autofs4 lock_dlm(U)
gfs(U) lock_harness(U) dlm(U) cman(U) md5 ipv6 sunrpc arpt_mangle
arptable_filter arp_tables dm_mirror dm_round_robin dm_multipath button
battery ac ohci_hcd tg3 bonding(U) floppy sg st ext3 jbd dm_mod qla2300
qla2xxx scsi_transport_fc cciss sd_mod scsi_mod
May 24 07:23:13 ancona kernel: CPU: 0
May 24 07:23:13 ancona kernel: EIP: 0060:[<f8ae8798>] Not tainted VLI
May 24 07:23:13 ancona kernel: EFLAGS: 00010246 (2.6.9-67.0.7.ELsmp)
May 24 07:23:13 ancona kernel: EIP is at do_dlm_lock+0x134/0x14e [lock_dlm]
May 24 07:23:13 ancona kernel: eax: 00000001 ebx: ffffffea ecx:
ee2f5c34 edx: f8aed2d3
May 24 07:23:13 ancona kernel: esi: f8ae87b7 edi: f7e4cc00 ebp:
e5fa9280 esp: ee2f5c30
May 24 07:23:13 ancona kernel: ds: 007b es: 007b ss: 0068
May 24 07:23:13 ancona kernel: Process httpd (pid: 10812,
threadinfo=ee2f5000 task=f4f6b830)
May 24 07:23:13 ancona kernel: Stack: f8aed2d3 20202020 32202020
20202020 20202020 32613720 36326365 f8b30018
May 24 07:23:13 ancona kernel: 00000246 e5fa9280 00000003
00000000 e5fa9280 f8ae8847 00000003 f8af0c80
May 24 07:23:13 ancona kernel: f8a53000 f8b48e9a 00000008
00000001 e78e546c e78e5450 f8a53000 f8b3ea9a
May 24 07:23:13 ancona kernel: Call Trace:
May 24 07:23:13 ancona kernel: [<f8b30018>]
gfs_acl_validate_set+0x18/0x8d [gfs]
May 24 07:23:13 ancona kernel: [<f8ae8847>] lm_dlm_lock+0x49/0x52
[lock_dlm]
May 24 07:23:13 ancona kernel: [<f8b48e9a>] gfs_lm_lock+0x35/0x4d [gfs]
May 24 07:23:13 ancona kernel: [<f8b3ea9a>]
gfs_glock_xmote_th+0x130/0x172 [gfs]
May 24 07:23:13 ancona kernel: [<f8b3e159>] rq_promote+0xc8/0x147 [gfs]
May 24 07:23:13 ancona kernel: [<f8b3e345>] run_queue+0x91/0xc1 [gfs]
May 24 07:23:13 ancona kernel: [<f8b3f355>] gfs_glock_nq+0xcf/0x116 [gfs]
May 24 07:23:13 ancona kernel: [<f8b3f92b>] gfs_glock_nq_init+0x13/0x26
[gfs]
May 24 07:23:13 ancona kernel: [<f8b42f44>] gfs_lookupi+0x321/0x3bf [gfs]
May 24 07:23:13 ancona kernel: [<f8b564c8>] gfs_lookup+0x83/0xfb [gfs]
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
