On Fri, Jun 24, 2022 at 08:13:09AM +1000, Dave Chinner wrote: > On Thu, Jun 23, 2022 at 02:52:22PM -0500, Clay Gerrard wrote: > > I work on an object storage system, OpenStack Swift, that has always > > used xfs on the storage nodes. Our system has encountered many > > various disk failures and occasionally apparent file system corruption > > over the years, but we've been noticing something lately that might be > > "new" and I'm considering how to approach the problem. I'm interested > > to solicit critique on my current thinking/process - particularly from > > xfs experts. > > > > [root@s8k-sjc3-d01-obj-9 ~]# xfs_bmap > > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53 > > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53: > > No data available > > [root@s8k-sjc3-d01-obj-9 ~]# xfs_db > > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53 > > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53: > > No data available > > ENODATA implies that it's trying to access an xattr that doesn't > exist. > > > fatal error -- couldn't initialize XFS library > > [root@s8k-sjc3-d01-obj-9 ~]# ls -alhF /srv/node/d21865/quarantined/objects-1/e53 > > ls: cannot access > > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53: > > No data available > > total 4.0K > > drwxr-xr-x 9 swift swift 318 Jun 7 00:57 ./ > > drwxr-xr-x 33 swift swift 4.0K Jun 23 16:10 ../ > > d????????? ? ? ? ? ? f0418758de4baaa402eb301c5bae3e53/ > > That's the typical ls output when it couldn't stat() an inode. This > typically occurs when the inode has been corrupted. On XFS, at > least, this should result in a corruption warning in the kernel log. > > Did you check dmesg for errors? Just to close the circle on the list thread - this is being caused by failing storage hardware, not filesystem issue. > [3600194.452706] sd 0:0:3:0: [sdd] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s > [3600194.452715] sd 0:0:3:0: [sdd] tag#5 Sense Key : Medium Error [current] [descriptor] > [3600194.452719] sd 0:0:3:0: [sdd] tag#5 Add. Sense: Unrecovered read error > [3600194.452723] sd 0:0:3:0: [sdd] tag#5 CDB: Read(16) 88 00 00 00 00 03 00 1c 76 30 00 00 00 20 00 00 > [3600194.452727] blk_update_request: critical medium error, dev sdd, sector 12886767168 > [3600194.453801] XFS (sdd): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x3001c7630 len 32 error 61 > [3600194.454846] XFS (sdd): xfs_imap_to_bp: xfs_trans_read_buf() returned error -61. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
