First, I know that I'm running a very old kernel. If this isn't the right
place to ask these questions, please let me know. I'm running BeeGFS (a
parallel disributed filesystem) on CentOS 7.9 systems (kernel
3.10.0-1160.76.1.el7.x86_64). On my metadata servers, I run XFS for the
filesystems holding the metadata. They run on top of software RAID1
mirrors on SSDs, are formatted like this:
mkfs.xfs --m crc=1,finobt=1 -i maxpct=95 -l size=400m /dev/md122
and are mounted with these options:
rw,noatime,nodiratime,attr2,nobarrier,inode64,noquota
A typical one of my metadata filesystems has ~210GB of space and
~400M inodes used. Last week, one of these filesystems shutdown with
these messages:
Nov 2 05:29:30 bmd3 kernel: XFS (md122): Internal error XFS_WANT_CORRUPTED_GOTO at line 3305 of file fs/xfs/libxfs/xfs_btree.c. Caller xfs_inobt_insert_rec+0x1f/0x30 [xfs]
Nov 2 05:29:30 bmd3 kernel: CPU: 28 PID: 11060 Comm: Worker9 Kdump: loaded Not tainted 3.10.0-1160.76.1.el7.x86_64 #1
Nov 2 05:29:30 bmd3 kernel: Hardware name: Supermicro Super Server/X11DDW-L, BIOS 3.1 04/30/2019
Nov 2 05:29:30 bmd3 kernel: Call Trace:
Nov 2 05:29:30 bmd3 kernel: [<ffffffffab1865c9>] dump_stack+0x19/0x1b
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06bc52b>] xfs_error_report+0x3b/0x40 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06a70bf>] ? xfs_inobt_insert_rec+0x1f/0x30 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc0694ddb>] xfs_btree_insert+0x1db/0x1f0 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06a70bf>] xfs_inobt_insert_rec+0x1f/0x30 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06a9f53>] xfs_difree_finobt+0xb3/0x200 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06aa1bb>] xfs_difree+0x11b/0x1d0 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06ce133>] xfs_ifree+0x83/0x150 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06ce2c8>] xfs_inactive_ifree+0xc8/0x230 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06ce4bb>] xfs_inactive+0x8b/0x130 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffc06d5b25>] xfs_fs_destroy_inode+0x95/0x190 [xfs]
Nov 2 05:29:30 bmd3 kernel: [<ffffffffaac6c61b>] destroy_inode+0x3b/0x60
Nov 2 05:29:30 bmd3 kernel: [<ffffffffaac6c755>] evict+0x115/0x180
Nov 2 05:29:30 bmd3 kernel: [<ffffffffaac6cb2c>] iput+0xfc/0x190
Nov 2 05:29:30 bmd3 kernel: [<ffffffffaac6097e>] do_unlinkat+0x1ae/0x2d0
Nov 2 05:29:30 bmd3 kernel: [<ffffffffaad98858>] ? lockref_put_or_lock+0x48/0x80
Nov 2 05:29:30 bmd3 kernel: [<ffffffffaac72454>] ? mntput+0x24/0x40
Nov 2 05:29:30 bmd3 kernel: [<ffffffffaac61a36>] SyS_unlink+0x16/0x20
Nov 2 05:29:30 bmd3 kernel: [<ffffffffab199f92>] system_call_fastpath+0x25/0x2a
Nov 2 05:29:30 bmd3 kernel: XFS (md122): xfs_inactive_ifree: xfs_ifree returned error -117
Nov 2 05:29:30 bmd3 kernel: XFS (md122): xfs_do_force_shutdown(0x1) called from line 1756 of file fs/xfs/xfs_inode.c. Return address = ffffffffc06ce353
Nov 2 05:29:31 bmd3 kernel: XFS (md122): I/O Error Detected. Shutting down filesystem
Nov 2 05:29:31 bmd3 kernel: XFS (md122): Please umount the filesystem and rectify the problem(s)
There were no other messages in the logs to indicate any sort of hardware
issue. After unmounting, I tried to run "xfs_repair" and it told me I
needed to mount to replay the log, unmount, then run "xfs_repair". Every
time I mounted, though, the filesystem shutdown with the same error. To
get the filesystem back online (and because the metadata is mirrored), I
ended up reformatting the drive.
In an only tangentially related effort, I'm working on a new backup
strategy for these filesystems using xfsdump rather than tar. I went to
run a level 0 dump of a similar filesystem on another server, and I saw
this:
# xfsdump -f /scratch/meta22.0.xfsd -l 0 -L meta22-0 -M meta22-0 /data/meta22
xfsdump: using file dump (drive_simple) strategy
xfsdump: version 3.1.7 (dump format 3.0) - type ^C for status and control
xfsdump: level 0 dump of bmd1.wynton.ucsf.edu:/data/meta22
xfsdump: dump date: Tue Nov 8 09:13:26 2022
xfsdump: session id: 022a0862-d5cb-41b9-88f2-9183f7821c86
xfsdump: session label: "meta22-0"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: skipping (only one dump stream)
xfsdump: ino map construction complete
xfsdump: estimated dump size: 129459315456 bytes
xfsdump: /var/lib/xfsdump/inventory created
xfsdump: creating dump session media file 0 (media 0, file 0)
xfsdump: dumping ino map
xfsdump: dumping directories
xfsdump: dumping non-directory files
xfsdump: WARNING: inomap inconsistency ino 472259: map says changed dir but is now non-dir: NOT dumping
xfsdump: WARNING: inomap inconsistency ino 1145788: map says changed dir but is now non-dir: NOT dumping
This is followed by many more lines like those last 2. Given these 2
issues so close together on such important systems, I'm a bit concerned.
Is there any hint as to what caused the filesystem crash? Are the "inomap
inconsistency" messages something to be concerned about? Are these issues
related in any way? Please let me know what other info I can provide, and
thanks for any help.
--
Joshua Baker-LePain
Wynton Cluster Sysadmin
UCSF
