Hi there. We have a large BeeGFS system here, and our metadata targets
use XFS. The metadata targets run on top of software RAID1 mirrors, and
BeeGFS mirrors each target across two servers. Our base OS is CentOS-7
(up to date). On Friday, the primary FS of a metadata pair crashed. We
successfully failed over to the secondary. Just a few minutes later,
though, that FS crashed with the same error message. We've been
attempting recovery. If you are or know a consultant skilled in XFS,
please have them contact me. Thank you so much.
For reference, the error message we got was this:
[14074823.591265] XFS (md125): Internal error XFS_WANT_CORRUPTED_GOTO at line 3305 of file fs/xfs/libxfs/xfs_btree.c. Caller xfs_inobt_insert_rec+0x1f/0x30 [xfs]
[14074823.606436] CPU: 43 PID: 173948 Comm: Worker34 Kdump: loaded Not tainted 3.10.0-1160.90.1.el7.x86_64 #1
[14074823.616581] Hardware name: Supermicro Super Server/X11DDW-L, BIOS 3.1 04/30/2019
[14074823.624695] Call Trace:
[14074823.627781] [<ffffffffa4bb1bec>] dump_stack+0x19/0x1f
[14074823.633655] [<ffffffffc073ebaf>] xfs_error_report+0x3f/0x50 [xfs]
[14074823.640550] [<ffffffffc07290cf>] ? xfs_inobt_insert_rec+0x1f/0x30 [xfs]
[14074823.647933] [<ffffffffc0716723>] xfs_btree_insert+0x1e3/0x1f0 [xfs]
[14074823.654992] [<ffffffffc07290cf>] xfs_inobt_insert_rec+0x1f/0x30 [xfs]
[14074823.662175] [<ffffffffc072c003>] xfs_difree_finobt+0xb3/0x200 [xfs]
[14074823.669215] [<ffffffffc072c273>] xfs_difree+0x123/0x1d0 [xfs]
[14074823.675690] [<ffffffffc0750c23>] xfs_ifree+0x83/0x150 [xfs]
[14074823.681989] [<ffffffffc0750db8>] xfs_inactive_ifree+0xc8/0x230 [xfs]
[14074823.689087] [<ffffffffc0750fab>] xfs_inactive+0x8b/0x140 [xfs]
[14074823.695688] [<ffffffffc0758805>] xfs_fs_destroy_inode+0x95/0x190 [xfs]
[14074823.702976] [<ffffffffa467aa4b>] destroy_inode+0x3b/0x70
[14074823.709067] [<ffffffffa467ab95>] evict+0x115/0x180
[14074823.714583] [<ffffffffa467af6c>] iput+0xfc/0x190
[14074823.719925] [<ffffffffa466e8f6>] do_unlinkat+0x1b6/0x2e0
[14074823.726024] [<ffffffffa4680c84>] ? mntput+0x24/0x40
[14074823.731660] [<ffffffffa466fa16>] SyS_unlink+0x16/0x20
[14074823.737456] [<ffffffffa4bc539a>] system_call_fastpath+0x25/0x2a
[14074823.744130] XFS (md125): xfs_inactive_ifree: xfs_ifree returned error -117
[14074823.751684] XFS (md125): xfs_do_force_shutdown(0x1) called from line 1756 of file fs/xfs/xfs_inode.c. Return address = ffffffffc0750e43
[14074823.795025] XFS (md125): I/O Error Detected. Shutting down filesystem
[14074823.802300] XFS (md125): Please umount the filesystem and rectify the problem(s)
--
Joshua Baker-LePain
Wynton Cluster Sysadmin
UCSF
