On Wed, 2008-03-26 at 10:05 +0000, DRand@xxxxxxxxxxx wrote: > > Hi Bob, > > Great, thanks for your info! The disk was previously a GFS disk and we > reformatted it with exactly the same mkfs command both times. Here are > more details. We are running the cluster on a Netapp SAN device. > > 1) mkfs.gfs -J 1024 -j 4 -p lock_gulm -t aicluster:cmsgfs /dev/sda > [100Gb device] > 2) Copy lots of files to the disk > 3) gfs_grow /san [Extra 50Gb extension added to device] > 4) Copy lots of files to the disk > 5) mkfs.gfs -J 1024 -j 4 -p lock_gulm -t aicluster:cmsgfs /dev/sda This is likely to be a problem. The first mkfs would have calculated where to put the resource groups based on the original size of the device. The gfs_grow would have added new resource groups, but it would calculate them based on the size of the extension. It uses a somewhat different set of calculations than mkfs, so the first "new" resource group added by gfs_grow will normally be in a location different from where mkfs would have put it, had the device been the new size to begin with. What that means is that the second mkfs would have recalculated new locations for the new RGs based on the bigger device size and put them in different locations. That would be likely to have overwritten some data from the old file system. If you were using lvm2 for your device, you may have been able to recover some data (with possible corruption interspersed from the second mkfs's RGs as stated above) by doing something like an lvremove of the bigger device, lvcreate of the 100Gb device, mkfs, lvresize +50G, then gfs_grow to place the RGs in their former locations. With a device name of /dev/sda, it sounds like you did not use lvm2, so I don't know if it's possible to re-shrink the gfs partition to its original size and re-grow it again. > I have now read about resource groups and the GFS ondisk structure > here.. > > http://www.redhat.com/archives/cluster-devel/2006-August/msg00324.html > > A couple more questions if you don't mind... > > What exactly would the mkfs command have done? Would the mkfs command > have overwritten the resource group headers from the previous disk > structure? Or does it just wipe the superblock and journals? The mkfs command does this: 1. Rewrite the superblock, which is no great loss. 2. Rewrite the internal "journal index" file which is not very destructive. 3. Rewrite the resource group index file (if the device has grown, the file might be slightly larger, so it may overwrite a few blocks). 4. Rewrite the root directory, which, if tricked into thinking there was valid data on all blocks, would cause gfs_fsck to toss all the files it finds into lost+found. So the root directory information would be lost. 5. Rewrite the quota file, which is no great loss. 6. Rewrite the license file, which is no great loss. The license file hasn't been used since Red Hat open-sourced the gfs code, but lately it's been re-used for the "fast statfs" feature. 7. Initializes the Resource Groups and the bitmaps that follow them. Again, if the file system had grown, these new RGs and bitmaps would be located in different places and overwrite blocks of data. 8. Initializes the journals to an empty state (again, if the device had grown, I guarantee if would put the new journals in a different place, thereby overwriting data from the first file system). > If the resource group headers still exist shouldn't they have a > characteristic structure we could identify enabling us to put 0xFF in > only the correct places on disk? The gfs2_edit tool will show the structure header in one color and the data in another. It will say something like "(rsrc grp hdr)" at the top of the screen for the RGs, and something like "(rsrc grp bitblk)" for the bitmap blocks. All the data that follows the headers for both structures are considered bits in the bitmap. The bitmaps are in the blocks immediately following the RGs. > Also is there anyway we can usefully depend on this information. Or > would mkfs have wiped these special inodes too? The internal files, or special inodes would have been rewritten by the mkfs. > In particular there is one 11Gb complete backup tar.gz on the disk > somewhere. I'm thinking if we could write some custom utility that > recognizes the gfs on disk structure and extracts very large files > from it? This sounds like a fun project. I would love it if I had time for such a thing. If you only wanted to try to recover that one file, this would likely be much safer than using gfs_mkfs, gfs_grow and gfs_fsck. You could start with the source to gfs2_edit and make it do a block-by-block search for gfs disk inodes, or gfs_dinode structures in each block. For every dinode found, you could read it in using the function gfs_dinode_in, then check to see if the di_size (file size) is close to what you want. If it finds the right file, you would have it mark that block in the bitmap as being "used metadata". Again, there are functions in libgfs2 to accomplish that. I'm not sure offhand if gfs_fsck would re-link the file into lost+found with only that much information. If not, you would have to create a directory entry for that dinode in the empty root directory. Then you'd want run gfs_fsck so it could find all the blocks from the file and mark them as "in use" as well. Still, some of your blocks may have been overwritten by bits from the new journals or new resource groups. If so, it will likely confuse gunzip and/or tar, and you'll still not be able to get at the data. But hey, you might get lucky. If you do decide to go this route, please send your changes to the public cluster-devel mailing list so everyone can benefit, okay? Regards, Bob Peterson Red Hat GFS -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
