We fscked the filesystem because we'd started seeing the following errors following a power failure. GFS: fsid=nearlineA:gfs1.0: fatal: invalid metadata block GFS: fsid=nearlineA:gfs1.0: bh = 2644310219 (type: exp=4, found=5) GFS: fsid=nearlineA:gfs1.0: function = gfs_get_meta_buffer GFS: fsid=nearlineA:gfs1.0: file = /usr/src/redhat/BUILD/gfs-kernel-2.6.9-49/smp/src/gfs/dio.c, line = 1223 GFS: fsid=nearlineA:gfs1.0: time = 1154425344 GFS: fsid=nearlineA:gfs1.0: about to withdraw from the cluster GFS: fsid=nearlineA:gfs1.0: waiting for outstanding I/O GFS: fsid=nearlineA:gfs1.0: telling LM to withdraw lock_dlm: withdraw abandoned memory GFS: fsid=nearlineA:gfs1.0: withdrawn And another instance: GFS: fsid=nearlineA:gfs1.1: fatal: filesystem consistency error GFS: fsid=nearlineA:gfs1.1: inode = 2384574146/2384574146 GFS: fsid=nearlineA:gfs1.1: function = dir_e_del GFS: fsid=nearlineA:gfs1.1: file = /usr/src/redhat/BUILD/gfs-kernel-2.6.9-49/smp/src/gfs/dir.c, line = 1495 GFS: fsid=nearlineA:gfs1.1: time = 1154393717 GFS: fsid=nearlineA:gfs1.1: about to withdraw from the cluster GFS: fsid=nearlineA:gfs1.1: waiting for outstanding I/O GFS: fsid=nearlineA:gfs1.1: telling LM to withdraw lock_dlm: withdraw abandoned memory GFS: fsid=nearlineA:gfs1.1: withdrawn Running gfs_fsck -vvv -y /dev/gfs1_vg/gfs1_lv Returns the following after chewing all the physical and swap RAM. The machines have 4Gb or RAM and 2Gb of swap. We can increase the swap size, but is this just gonna keep running out of RAM? We're running on x86_64 so it can use as much memory as it likes. The filesystem is roughly 45Tb. Initializing fsck Initializing lists... Initializing special inodes... Setting block ranges... Creating a block list of size 11105160192... Unable to allocate bitmap of size 1388145025 Segmentation fault [root@ns1a ~]# gfs_fsck -vvv -y /dev/gfs1_vg/gfs1_lv Initializing fsck Initializing lists... (bio.c:140) Writing to 65536 - 16 4096 Initializing special inodes... (file.c:45) readi: Offset (640) is >= the file size (640). (super.c:208) 8 journals found. (file.c:45) readi: Offset (7116576) is >= the file size (7116576). (super.c:265) 74131 resource groups found. Setting block ranges... Creating a block list of size 11105160192... (bitmap.c:68) Allocated bitmap of size 5552580097 with 2 chunks per byte Unable to allocate bitmap of size 1388145025 (block_list.c:72) <backtrace> - block_list_create() Segmentation fault -- Stephen Willey Senior Systems Engineer, Framestore-CFC +44 (0)207 344 8000 http://www.framestore-cfc.com -- Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
