Hardware
Configuration: Six node cluster, each
node has a LSI Fibre Channel Host Adapter interface to a
SAN.
Software
Configuration: The kernel is 2.4.21-20.EL
with GFS-6.0.2-25
Problem: While four nodes are
simultaneously accessing the SAN, if a 5th node attempts to access the SAN, one
of the nodes will kernel panic.
The node that crashes seems to be random. All the crashes have the same
error as follows:
May 17 21:53:52 compute-0-2.local kernel:
mptscsih: ioc0: WARNING - Device (0:0:1) reported QUEUE_FULL!
May 17 21:53:52 compute-0-2.local kernel:
SCSI disk error : host 0 channel 0 id 0 lun 1 return code = 440b0000
May 17 21:53:52 compute-0-2.local kernel:
I/O error: dev 08:12, sector 139961968
May 17 21:53:52 compute-0-2.local kernel:
Pool: IO request to device, (8,18) blk #139961968, failed.
May 17 21:53:52 compute-0-2.local kernel:
GFS: fsid=p2-2:gfs1.3: read error on block 17495244
May 17 21:53:52 compute-0-2.local kernel:
Panicking because of read error on block 17495244
May 17 21:53:52 compute-0-2.local kernel:
f3d33b98 f8a2f2a2 00000032 00000031 c01217d2 0000000a 00000400 f8a4f7f5
May 17 21:53:52 compute-0-2.local kernel:
f3d33be8 f3740370 010af4cc f3740370 00000020 00000000 f8a5c000 00000031
May 17 21:53:52 compute-0-2.local kernel:
f8a1419e f8a4d692 f8a4d57a 0000024f 00000013 f8a5c000 f8a5c000 f3d33c3c
May 17 21:53:52 compute-0-2.local kernel:
Call Trace: [<f8a2f2a2>] gfs_asserti [gfs] 0x32 (0xf3d33b9c)
May 17 21:53:52 compute-0-2.local kernel:
[<c01217d2>] printk [kernel] 0x122 (0xf3d33ba8)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a4f7f5>] .rodata.str1.4 [gfs] 0x249 (0xf3d33bb4)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a1419e>] gfs_dreread [gfs] 0x12e (0xf3d33bd8)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a4d692>] .rodata.str1.1 [gfs] 0x1e6 (0xf3d33bdc)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a4d57a>] .rodata.str1.1 [gfs] 0xce (0xf3d33be0)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a13ff9>] gfs_dread [gfs] 0x49 (0xf3d33bfc)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a1513f>] gfs_get_meta_buffer [gfs] 0x9f (0xf3d33c18)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a22fc2>] get_metablock [gfs] 0xb2 (0xf3d33c50)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a233db>] gfs_block_map [gfs] 0x2eb (0xf3d33c70)
May 17 21:53:52 compute-0-2.local kernel:
[<c016ad48>] init_buffer_head [kernel] 0x38 (0xf3d33cb8)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a1cdae>] get_block [gfs] 0x9e (0xf3d33d28)
May 17 21:53:52 compute-0-2.local kernel:
[<c01567db>] __block_prepare_write [kernel] 0x19b (0xf3d33d64)
May 17 21:53:52 compute-0-2.local kernel:
[<c014a0d0>] __alloc_pages_limit [kernel] 0x60 (0xf3d33d94)
May 17 21:53:52 compute-0-2.local kernel:
[<c0157139>] block_prepare_write [kernel] 0x39 (0xf3d33da8)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a1cd10>] get_block [gfs] 0x0 (0xf3d33dbc)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a1d41c>] gfs_prepare_write [gfs] 0x11c (0xf3d33dc8)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a1cd10>] get_block [gfs] 0x0 (0xf3d33dd8)
May 17 21:53:52 compute-0-2.local kernel:
[<c013f5d5>] do_generic_file_write [kernel] 0x1d5 (0xf3d33df0)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a1790b>] do_do_write [gfs] 0x2ab (0xf3d33e44)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a17d4b>] do_write [gfs] 0x18b (0xf3d33e90)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a15c89>] gfs_walk_vma [gfs] 0x129 (0xf3d33ecc)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a1f112>] gfs_sync_page [gfs] 0x52 (0xf3d33eec)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a319b7>] gfs_glock_nq_init [gfs] 0x37 (0xf3d33f30)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a319f3>] gfs_glock_dq_uninit [gfs] 0x13 (0xf3d33f40)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a187e1>] gfs_sync_file [gfs] 0x61 (0xf3d33f4c)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a17e20>] gfs_write [gfs] 0x90 (0xf3d33f6c)
May 17 21:53:52 compute-0-2.local kernel:
[<f8a17bc0>] do_write [gfs] 0x0 (0xf3d33f80)
May 17 21:53:52 compute-0-2.local kernel:
[<c0153a53>] sys_write [kernel] 0xa3 (0xf3d33f94)
May 17 21:53:52 compute-0-2.local kernel:
May 17 21:53:52 compute-0-2.local kernel:
Kernel panic: GFS: Assertion failed on line 591 of file linux_dio.c
May 17 21:53:52 compute-0-2.local kernel:
GFS: assertion: "FALSE"
May 17 21:53:52 compute-0-2.local kernel:
GFS: time = 1116388432
May 17 21:53:52 compute-0-2.local kernel:
GFS: fsid=p2-2:gfs1.3
May 17 21:53:52 compute-0-2.local kernel:
Frank L. Setinsek
-- Linux-cluster@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/linux-cluster
