Every few days or so our cluster machines seem to have kernel panics
comp laing about GFS locking (although its pretty irregular, we went for
a few weeks without an outage)
We noticed that this happened a LOT, and it was reproducible when
certain users accessed files, when we were serving afp off the cluster.
We have changed things since then so that afp is run on a server which
nfs mounts the cluster.
We are running FC4 with the gfs modules from yum.
Here is our most recent kernel panics, followed by one from when we had
afp running on the cluster: (it looks like there is relevant info above
the cut-here, possibly if it might be helpful)
Oct 19 14:44:41 meow kernel: ------------[ cut here ]------------
Oct 19 14:44:41 meow kernel: kernel BUG at
/usr/src/build/607755-i686/BUILD/smp/src/lockqueue.c:1144!
Oct 19 14:44:41 meow kernel: invalid operand: 0000 [#1]
Oct 19 14:44:41 meow kernel: SMP
Oct 19 14:44:41 meow kernel: Modules linked in: nfsd exportfs lockd
autofs4 lock_dlm(U) gfs(U) lock_harness(U) rfcomm l2cap bluetooth dlm(U)
cman(U) md5 ip
v6 sunrpc ipt_LOG ipt_limit ipt_state ip_conntrack iptable_filter
ip_tables video button battery ac uhci_hcd ehci_hcd hw_random i2c_i801
i2c_core shpchp e1000 floppy ext3 jbd raid1 dm_mod qla2200 qla2xxx
scsi_transport_fc ata_piix libata sd_mod scsi_mod
Oct 19 14:44:41 meow kernel: CPU: 1
Oct 19 14:44:41 meow kernel: EIP: 0060:[<f8af9dcf>] Not tainted VLI
Oct 19 14:44:41 meow kernel: EFLAGS: 00010292 (2.6.12-1.1447_FC4smp)
Oct 19 14:44:41 meow kernel: EIP is at
process_cluster_request+0xddb/0xdef [dlm]
Oct 19 14:44:41 meow kernel: eax: 00000004 ebx: 00000000 ecx:
c035fa4c edx: 00000286
Oct 19 14:44:41 meow kernel: esi: f7fb8400 edi: 00000000 ebp:
d2988000 esp: f7eefe24
Oct 19 14:44:41 meow kernel: ds: 007b es: 007b ss: 0068
Oct 19 14:44:41 meow kernel: Process dlm_recvd (pid: 2402,
threadinfo=f7eef000 task=f7851020)
Oct 19 14:44:41 meow kernel: Stack: f8b0621b 00000001 f8b071e0 f8b06217
2583f987 00000001 00000040 00004000
Oct 19 14:44:41 meow kernel: f7eefe48 00000000 c038e1a0 00000a58
f0167b00 c02a26c1 00000a58 00004040
Oct 19 14:44:41 meow kernel: 00000072 f7eefed4 00000000 00000001
00000246 00000000 edd6eeb8 00000000
Oct 19 14:44:41 meow kernel: Call Trace:
Oct 19 14:44:41 meow kernel: [<c02a26c1>] sock_recvmsg+0x103/0x11e
Oct 19 14:44:41 meow kernel: [<f8afd46b>]
midcomms_process_incoming_buffer+0x13b/0x25f [dlm]
Oct 19 14:44:41 meow kernel: [<c011ce54>] load_balance_newidle+0x23/0x82
Oct 19 14:44:41 meow kernel: [<f8afb3d3>] receive_from_sock+0x196/0x2c9
[dlm]
Oct 19 14:44:41 meow kernel: [<c0307705>] schedule+0x405/0xc5e
Oct 19 14:44:41 meow kernel: [<c0307731>] schedule+0x431/0xc5e
Oct 19 14:44:41 meow kernel: [<f8afc457>] dlm_recvd+0x0/0x9c [dlm]
Oct 19 14:44:41 meow kernel: [<f8afc2d3>] process_sockets+0x75/0xb7 [dlm]
Oct 19 14:44:41 meow kernel: [<f8afc4c7>] dlm_recvd+0x70/0x9c [dlm]
Oct 19 14:44:41 meow kernel: [<c0134c09>] kthread+0x93/0x97
Oct 19 14:44:41 meow kernel: [<c0134b76>] kthread+0x0/0x97
Oct 19 14:44:41 meow kernel: [<c01023d1>] kernel_thread_helper+0x5/0xb
Oct 19 14:44:41 meow kernel: Code: 4f 82 62 c7 89 e8 e8 b1 b4 00 00 8b
4c 24 14 89 4c 24 04 c7 04 24 6d 63 b0 f8 e8 34 82 62 c7 c7 04 24 1b 62
b0 f8 e8 28
82 62 c7 <0f> 0b 78 04 e0 71 b0 f8 c7 04 24 70 72 b0 f8 e8 40 78 62 c7 57
Oct 19 14:44:41 meow kernel: <0>Fatal exception: panic in 5 seconds
Panic 2:
Oct 10 09:58:39 woof kernel: ------------[ cut here ]------------
Oct 10 09:58:39 woof kernel: kernel BUG at
/usr/src/build/607778-i686/BUILD/smp/src/dlm/lock.c:411!
Oct 10 09:58:39 woof kernel: invalid operand: 0000 [#1]
Oct 10 09:58:39 woof kernel: SMP
Oct 10 09:58:39 woof kernel: Modules linked in: nfsd exportfs lockd
autofs4 lock_dlm(U) gfs(U) lock_harness(U) rfcomm l2cap bluetooth dlm(U)
cman(U) md5 ip
v6 sunrpc ipt_LOG ipt_limit ipt_state ip_conntrack iptable_filter
ip_tables video button battery ac uhci_hcd ehci_hcd hw_random i2c_i801
i2c_core shpchp e1
000 dm_snapshot dm_zero dm_mirror ext3 jbd raid1 dm_mod qla2200 qla2xxx
scsi_transport_fc ata_piix libata sd_mod scsi_mod
Oct 10 09:58:39 woof kernel: CPU: 1
Oct 10 09:58:39 woof kernel: EIP: 0060:[<f8b98bf5>] Not tainted VLI
Oct 10 09:58:39 woof kernel: EFLAGS: 00010292 (2.6.12-1.1447_FC4smp)
Oct 10 09:58:39 woof kernel: EIP is at do_dlm_lock+0x1b7/0x21d [lock_dlm]
Oct 10 09:58:39 woof kernel: eax: 00000004 ebx: 00000000 ecx:
c035fa4c edx: 00000292
Oct 10 09:58:39 woof kernel: esi: f7848140 edi: ffffffea ebp:
00000003 esp: c74b3cfc
Oct 10 09:58:39 woof kernel: ds: 007b es: 007b ss: 0068
Oct 10 09:58:39 woof kernel: Process imapd (pid: 24278,
threadinfo=c74b3000 task=f4721a80)
Oct 10 09:58:39 woof kernel: Stack: f8b9de75 f7848140 00000003 1bbe0000
00000000 ffffffea 00000003 00000005
Oct 10 09:58:39 woof kernel: 0000000d 00000005 00000000 f58c0a00
00000001 0000000d 20200000 20202020
Oct 10 09:58:39 woof kernel: 20203320 20202020 62312020 30306562
00183030 c8fb2f00 00000001 00000001
Oct 10 09:58:39 woof kernel: Call Trace:
Oct 10 09:58:39 woof kernel: [<f8b98cff>] lm_dlm_lock+0x52/0x5e [lock_dlm]
Oct 10 09:58:39 woof kernel: [<f8b98cad>] lm_dlm_lock+0x0/0x5e [lock_dlm]
Oct 10 09:58:39 woof kernel: [<f8bd000c>] gfs_lm_lock+0x3d/0x5c [gfs]
Oct 10 09:58:39 woof kernel: [<f8bc5039>] gfs_glock_xmote_th+0xae/0x1d3
[gfs]
Oct 10 09:58:39 woof kernel: [<f8bc463c>] rq_promote+0x126/0x150 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bc4840>] run_queue+0xee/0x113 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bc5af1>] gfs_glock_nq+0x93/0x144 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bc619d>] gfs_glock_nq_init+0x18/0x2d [gfs]
Oct 10 09:58:39 woof kernel: [<f8be3926>] get_local_rgrp+0xca/0x1b0 [gfs]
Oct 10 09:58:39 woof kernel: [<f8be3a9c>]
gfs_inplace_reserve_i+0x90/0xd0 [gfs]
Oct 10 09:58:39 woof kernel: [<f8be046b>] gfs_quota_lock_m+0xbf/0x117 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bd8a2b>] do_do_write_buf+0x3a1/0x485 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bc56a1>]
glock_wait_internal+0x16b/0x26a [gfs]
Oct 10 09:58:39 woof kernel: [<f8bd8c91>] do_write_buf+0x182/0x1b6 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bd7be5>] walk_vm+0xb3/0x111 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bd8d65>] gfs_write+0xa0/0xc2 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bd8b0f>] do_write_buf+0x0/0x1b6 [gfs]
Oct 10 09:58:39 woof kernel: [<f8bd8cc5>] gfs_write+0x0/0xc2 [gfs]
Oct 10 09:58:39 woof kernel: [<c0162987>] vfs_write+0x9e/0x110
Oct 10 09:58:39 woof kernel: [<c0162aa4>] sys_write+0x41/0x6a
Oct 10 09:58:39 woof kernel: [<c0104035>] syscall_call+0x7/0xb
Oct 10 09:58:39 woof kernel: Code: 7c 24 14 89 4c 24 0c 89 5c 24 10 89
6c 24 08 89 74 24 04 c7 04 24 28 e6 b9 f8 e8 0e 94 58 c7 c7 04 24 75 de
b9 f8 e8 02
94 58 c7 <0f> 0b 9b 01 a0 e4 b9 f8 c7 04 24 3c e5 b9 f8 e8 1a 8a 58 c7 66
Oct 10 09:58:39 woof kernel: <0>Fatal exception: panic in 5 seconds
Sep 7 15:37:44 meow kernel: ------------[ cut here ]------------
Sep 7 15:37:44 meow kernel: kernel BUG at
/usr/src/build/588748-i686/BUILD/smp/src/dlm/plock.c:500!
Sep 7 15:37:44 meow kernel: invalid operand: 0000 [#1]
Sep 7 15:37:44 meow kernel: SMP
Sep 7 15:37:44 meow kernel: Modules linked in: appletalk nfsd exportfs
lockd autofs4 lock_dlm(U) gfs(U) lock_harness(U) rfcomm l2cap bluetooth
dlm(U) cman
(U) sunrpc md5 ipv6 ipt_LOG ipt_limit ipt_state ip_conntrack
iptable_filter ip_tables video button battery ac uhci_hcd ehci_hcd
hw_random i2c_i801 i2c_core
shpchp e1000 floppy ext3 jbd raid1 dm_mod qla2200 qla2xxx
scsi_transport_fc ata_piix libata sd_mod scsi_mod
Sep 7 15:37:44 meow kernel: CPU: 3
Sep 7 15:37:44 meow kernel: EIP: 0060:[<f8b9a3f7>] Tainted:
GF VLI
Sep 7 15:37:44 meow kernel: EFLAGS: 00010292 (2.6.12-1.1398_FC4smp)
Sep 7 15:37:44 meow kernel: EIP is at update_lock+0x87/0x9b [lock_dlm]
Sep 7 15:37:44 meow kernel: eax: 00000004 ebx: fffffff5 ecx:
c035ca4c edx: 00000282
Sep 7 15:37:44 meow kernel: esi: 00000000 edi: e99c2c00 ebp:
00000000 esp: d05dedb4
Sep 7 15:37:44 meow kernel: ds: 007b es: 007b ss: 0068
Sep 7 15:37:44 meow kernel: Process afpd (pid: 3872,
threadinfo=d05de000 task=d6447550)
Sep 7 15:37:44 meow kernel: Stack: badc0ded f8b9d0d6 fffffff5 f8b9da70
f8b9d101 06609291 f7943000 00000000
Sep 7 15:37:44 meow kernel: f8b9a499 7ffffff8 00000000 7ffffff8
00000000 d05dede8 d7636700 7ffffff8
Sep 7 15:37:44 meow kernel: 00000000 d05deea8 d05dee28 f8b9a987
00000001 7ffffff8 00000000 7ffffff8
Sep 7 15:37:44 meow kernel: Call Trace:
Sep 7 15:37:44 meow kernel: [<f8b9a499>] add_lock+0x8e/0xed [lock_dlm]
Sep 7 15:37:44 meow kernel: [<f8b9a987>] fill_gaps+0x87/0x10e [lock_dlm]
Sep 7 15:37:44 meow kernel: [<f8b9aa51>] lock_case3+0x43/0xac [lock_dlm]
Sep 7 15:37:44 meow kernel: [<f8b9aeac>] plock_internal+0x1aa/0x370
[lock_dlm]
Sep 7 15:37:44 meow kernel: [<f8b9b614>] lm_dlm_plock+0x25b/0x2dc
[lock_dlm]
Sep 7 15:37:44 meow kernel: [<f8b9b3b9>] lm_dlm_plock+0x0/0x2dc [lock_dlm]
Sep 7 15:37:44 meow kernel: [<f8bdc1c3>] gfs_lm_plock+0x45/0x57 [gfs]
Sep 7 15:37:44 meow kernel: [<f8be5731>] gfs_lock+0xcd/0x11c [gfs]
Sep 7 15:37:44 meow kernel: [<f8be5664>] gfs_lock+0x0/0x11c [gfs]
Sep 7 15:37:44 meow kernel: [<c0176c4f>] fcntl_setlk64+0x16c/0x26a
Sep 7 15:37:44 meow kernel: [<c0162e93>] fget+0x3b/0x42
Sep 7 15:37:44 meow kernel: [<c0172bfd>] sys_fcntl64+0x55/0x97
Sep 7 15:37:44 meow kernel: [<c0104025>] syscall_call+0x7/0xb
Sep 7 15:37:44 meow kernel: Code: 01 00 00 c7 04 24 a8 da b9 f8 e8 7c
77 58 c7 89 5c 24 04 c7 04 24 08 d1 b9 f8 e8 6c 77 58 c7 c7 04 24 d6 d0
b9 f8 e8 60
77 58 c7 <0f> 0b f4 01 70 da b9 f8 c7 04 24 10 db b9 f8 e8 78 6d 58 c7 55
Sep 7 15:37:44 meow kernel: <0>Fatal exception: panic in 5 seconds
Thanks for any help,
Ethan
--
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
