On 11/21/11 12:06 PM, Gregory Farnum wrote: > While working with a Ceph node running XFS we somehow managed to > corrupt our filesystem. I don't think there were any hard powercycles > on this node, but while starting up after a kernel upgrade (it's > running 3.1) the daemon was performing its usual startup sequence (a > lot of file truncates, mostly) when it got an error out of the > filesystem: Others have had good comments but also: > 2011-11-17 16:00:37.294876 7f83f3eef720 filestore(/mnt/osd.17) > truncate meta/pginfo_12.7c8/0 size 0 > 2011-11-17 16:00:37.483407 7f83f3eef720 filestore(/mnt/osd.17) > truncate meta/pginfo_12.7c8/0 size 0 = -117 > 2011-11-17 16:00:37.483476 7f83f3eef720 filestore(/mnt/osd.17) error > error 117: Structure needs cleaning not handled was there anything in dmesg/system logs right at this point? XFS should have said something about this original error. -Eric > When I tried to look at the filesystem, it failed with EIO. When I > tried to mount the filesystem after a remount, it gave me an internal > error: > > root@cephstore6358:~# mount /dev/sdg1 /mnt/osd.17 > 2011 Nov 18 14:52:47 cephstore6358 [82374.729383] XFS: Internal error > XFS_WANT_CORRUPTED_GOTO at line 1664 of file fs/xfs/xfs_alloc.c. > Caller 0xffffffff811d6b71 > 2011 Nov 18 14:52:47 cephstore6358 [82374.729386] > 2011 Nov 18 14:52:47 cephstore6358 [82374.758262] XFS (sdg1): Internal > error xfs_trans_cancel at line 1928 of file fs/xfs/xfs_trans.c. > Caller 0xffffffff811fa463 > 2011 Nov 18 14:52:47 cephstore6358 [82374.758265] > 2011 Nov 18 14:52:47 cephstore6358 [82374.758352] XFS (sdg1): > Corruption of in-memory data detected. Shutting down filesystem > 2011 Nov 18 14:52:47 cephstore6358 [82374.758356] XFS (sdg1): Please > umount the filesystem and rectify the problem(s) > 2011 Nov 18 14:52:47 cephstore6358 [82374.758364] XFS (sdg1): Failed > to recover EFIs > mount: Structure needs cleaning > > dmesg had a little more output: > > dmesg says: > [82373.779312] XFS (sdg1): Mounting Filesystem > [82373.930531] XFS (sdg1): Starting recovery (logdev: internal) > [82374.729383] XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line > 1664 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff811d6b71 > [82374.729386] > [82374.741959] Pid: 30648, comm: mount Not tainted > 3.1.0-dho-00004-g1ffcb5c-dirty #1 > [82374.749543] Call Trace: > [82374.751994] [<ffffffff811d606e>] ? xfs_free_ag_extent+0x4e3/0x698 > [82374.758157] [<ffffffff811ce1f8>] ? xfs_setup_devices+0x84/0x84 > [82374.758163] [<ffffffff811ce1f8>] ? xfs_setup_devices+0x84/0x84 > [82374.758167] [<ffffffff811d6b71>] ? xfs_free_extent+0xb6/0xf9 > [82374.758171] [<ffffffff811d3034>] ? kmem_zone_alloc+0x58/0x9e > [82374.758179] [<ffffffff812095f9>] ? xfs_trans_get_efd+0x21/0x2a > [82374.758185] [<ffffffff811fa413>] ? xlog_recover_process_efi+0x113/0x172 > [82374.758190] [<ffffffff811fa54b>] ? xlog_recover_process_efis+0x4e/0x8e > [82374.758194] [<ffffffff811faa53>] ? xlog_recover_finish+0x14/0x88 > [82374.758199] [<ffffffff8120088e>] ? xfs_mountfs+0x46c/0x56a > [82374.758204] [<ffffffff811ce365>] ? xfs_fs_fill_super+0x16d/0x244 > [82374.758213] [<ffffffff810d5dcf>] ? mount_bdev+0x13d/0x198 > [82374.758218] [<ffffffff810d4a42>] ? mount_fs+0xc/0xa6 > [82374.758225] [<ffffffff810eb274>] ? vfs_kern_mount+0x61/0x97 > [82374.758230] [<ffffffff810eb316>] ? do_kern_mount+0x49/0xd6 > [82374.758234] [<ffffffff810eba99>] ? do_mount+0x6f6/0x75d > [82374.758241] [<ffffffff810b4429>] ? memdup_user+0x3a/0x56 > [82374.758246] [<ffffffff810ebb88>] ? sys_mount+0x88/0xc4 > [82374.758254] [<ffffffff8166c07b>] ? system_call_fastpath+0x16/0x1b > [82374.758262] XFS (sdg1): Internal error xfs_trans_cancel at line > 1928 of file fs/xfs/xfs_trans.c. Caller 0xffffffff811fa463 > > [82374.758265] > [82374.758268] Pid: 30648, comm: mount Not tainted > 3.1.0-dho-00004-g1ffcb5c-dirty #1 > [82374.758270] Call Trace: > [82374.758275] [<ffffffff81201ecd>] ? xfs_trans_cancel+0x56/0xcf > [82374.758279] [<ffffffff811fa463>] ? xlog_recover_process_efi+0x163/0x172 > [82374.758284] [<ffffffff811fa54b>] ? xlog_recover_process_efis+0x4e/0x8e > [82374.758288] [<ffffffff811faa53>] ? xlog_recover_finish+0x14/0x88 > [82374.758293] [<ffffffff8120088e>] ? xfs_mountfs+0x46c/0x56a > [82374.758298] [<ffffffff811ce365>] ? xfs_fs_fill_super+0x16d/0x244 > [82374.758303] [<ffffffff810d5dcf>] ? mount_bdev+0x13d/0x198 > [82374.758307] [<ffffffff810d4a42>] ? mount_fs+0xc/0xa6 > [82374.758312] [<ffffffff810eb274>] ? vfs_kern_mount+0x61/0x97 > [82374.758317] [<ffffffff810eb316>] ? do_kern_mount+0x49/0xd6 > [82374.758321] [<ffffffff810eba99>] ? do_mount+0x6f6/0x75d > [82374.758325] [<ffffffff810b4429>] ? memdup_user+0x3a/0x56 > [82374.758330] [<ffffffff810ebb88>] ? sys_mount+0x88/0xc4 > [82374.758335] [<ffffffff8166c07b>] ? system_call_fastpath+0x16/0x1b > [82374.758341] XFS (sdg1): xfs_do_force_shutdown(0x8) called from line > 1929 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff81201ee6 > [82374.758352] XFS (sdg1): Corruption of in-memory data detected. > Shutting down filesystem > [82374.758356] XFS (sdg1): Please umount the filesystem and rectify > the problem(s) > [82374.758364] XFS (sdg1): Failed to recover EFIs > [82374.758367] XFS (sdg1): log mount finish failed > > xfs_check doesn't give me much either, since I assume the errors above > are involved in log replay: > root@cephstore6358:~# xfs_check -v /dev/sdg1 > ERROR: The filesystem has valuable metadata changes in a log which needs to > be replayed. Mount the filesystem to replay the log, and unmount it before > re-running xfs_check. If you are unable to mount the filesystem, then use > the xfs_repair -L option to destroy the log and attempt a repair. > Note that destroying the log may cause corruption -- please attempt a mount > of the filesystem before doing this. > > Is there something useful I can do about this? Data I can provide to > help track down what broke? > -Greg > > _______________________________________________ > xfs mailing list > xfs@xxxxxxxxxxx > http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
