Hi all,
Sorry, I'm mailing here as a last resort before declaring this
filesystem done for. Following a string of unclean reboots and a dying
hard disk I have this filesystem in a very poor state that xfs_repair
can't make any progress on.
It has been mounted on kernel 5.18.14-1~bpo11+1 (from Debian
bullseye-backports). Most of the repairs were done using xfsprogs
5.10.0-4 (from Debian bullseye stable), though I did also try with
6.0.0-1 (from Debian bookworm/testing re-built myself).
I've attached the full log from xfs_repair, but the summary is it all
starts with multiple instances of this in Phase 3:
Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block
0xe101f32f8/0x1000
bad directory block magic # 0x1859dc06 in block 0 for directory inode
64426557977
bad bestfree table in block 0 in directory inode 64426557977: repairing
table
As it is the filesystem can be mounted and most data appears accessible,
but several directories are corrupt and can't be read or removed; the
kernel reports metadata corruption and CRC errors and returns EUCLEAN.
Ideally I'd like to remove the corrupt directories, recover as much of
what's left as possible, and make the filesystem usable again (it's an
rsnapshot destination) - but I'll take what I can.
Many thanks in advance,
Chris
PS: Please Cc me in replies
--
Chris Boot
bootc@xxxxxx
Phase 1 - find and verify superblock...
- reporting progress in intervals of 15 minutes
Phase 2 - using internal log
- zero log...
- 20:25:20: zeroing log - 521728 of 521728 blocks done
- scan filesystem freespace and inode maps...
- 20:25:23: scanning filesystem freespace - 38 of 38 allocation groups done
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- 20:25:23: scanning agi unlinked lists - 38 of 38 allocation groups done
- process known inodes and perform inode discovery...
- agno = 15
- agno = 0
- agno = 30
Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block 0xe101f32f8/0x1000
bad directory block magic # 0x1859dc06 in block 0 for directory inode 64426557977
bad bestfree table in block 0 in directory inode 64426557977: repairing table
- agno = 16
- agno = 31
- agno = 1
- agno = 17
- agno = 32
- agno = 2
Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block 0x7f8007bf8/0x1000
bad directory block magic # 0x92b36c92 in block 0 for directory inode 36507254400
bad bestfree table in block 0 in directory inode 36507254400: repairing table
- agno = 18
- agno = 33
Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block 0xf77fff228/0x1000
bad directory block magic # 0xd9ac4ca5 in block 0 for directory inode 70866962823
bad bestfree table in block 0 in directory inode 70866962823: repairing table
- agno = 3
- agno = 34
- agno = 19
Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block 0xff00332e0/0x1000
bad directory block magic # 0xa03871ec in block 0 for directory inode 73014585868
bad bestfree table in block 0 in directory inode 73014585868: repairing table
- agno = 4
- agno = 35
- agno = 20
Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block 0x960000640/0x1000
bad directory block magic # 0x10a5bf9c in block 0 for directory inode 42949675715
bad bestfree table in block 0 in directory inode 42949675715: repairing table
- agno = 5
- agno = 36
- agno = 21
- agno = 37
- agno = 6
- agno = 22
- agno = 7
- agno = 23
- agno = 8
- agno = 24
- agno = 9
Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block 0xb4002d4f0/0x1000
bad directory block magic # 0x9214d055 in block 0 for directory inode 51539795872
bad bestfree table in block 0 in directory inode 51539795872: repairing table
- agno = 25
- agno = 10
- agno = 26
- agno = 11
- agno = 27
Metadata CRC error detected at 0x5609236ce178, xfs_dir3_block block 0xca7fff978/0x1000
bad directory block magic # 0x5b58346c in block 0 for directory inode 57982059162
bad bestfree table in block 0 in directory inode 57982059162: repairing table
- agno = 12
- agno = 28
- agno = 13
- agno = 29
- agno = 14
- 20:32:26: process known inodes and inode discovery - 32678656 of 32678656 inodes done
- process newly discovered inodes...
- 20:32:26: process newly discovered inodes - 38 of 38 allocation groups done
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- 20:32:27: setting up duplicate extent list - 38 of 38 allocation groups done
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 2
- agno = 4
- agno = 7
- agno = 10
- agno = 3
- agno = 11
- agno = 8
- agno = 13
- agno = 24
- agno = 17
- agno = 1
- agno = 9
- agno = 14
- agno = 22
- agno = 35
- agno = 30
- agno = 12
- agno = 29
- agno = 27
- agno = 25
- agno = 28
- agno = 26
- agno = 23
- agno = 32
- agno = 20
bad directory block magic # 0x5b58346c in block 0 for directory inode 57982059162
- agno = 36
bad bestfree table in block 0 in directory inode 57982059162: repairing table
- agno = 6
- agno = 33
- agno = 34
- agno = 21
- agno = 37
- agno = 5
- agno = 19
- agno = 16
bad directory block magic # 0x10a5bf9c in block 0 for directory inode 42949675715
bad bestfree table in block 0 in directory inode 42949675715: repairing table
- agno = 31
- agno = 15
- agno = 18
bad directory block magic # 0xd9ac4ca5 in block 0 for directory inode 70866962823
bad bestfree table in block 0 in directory inode 70866962823: repairing table
bad directory block magic # 0x92b36c92 in block 0 for directory inode 36507254400
bad bestfree table in block 0 in directory inode 36507254400: repairing table
bad directory block magic # 0x9214d055 in block 0 for directory inode 51539795872
bad bestfree table in block 0 in directory inode 51539795872: repairing table
bad directory block magic # 0xa03871ec in block 0 for directory inode 73014585868
bad bestfree table in block 0 in directory inode 73014585868: repairing table
bad directory block magic # 0x1859dc06 in block 0 for directory inode 64426557977
bad bestfree table in block 0 in directory inode 64426557977: repairing table
- 20:32:35: check for inodes claiming duplicate blocks - 32678656 of 32678656 inodes done
Phase 5 - rebuild AG headers and trees...
- 20:32:37: rebuild AG headers and trees - 38 of 38 allocation groups done
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
bad directory block magic # 0x1859dc06 for directory inode 64426557977 block 0: fixing magic # to 0x58444233
bad directory block magic # 0x92b36c92 for directory inode 36507254400 block 0: fixing magic # to 0x58444233
bad directory block magic # 0xd9ac4ca5 for directory inode 70866962823 block 0: fixing magic # to 0x58444233
bad directory block magic # 0xa03871ec for directory inode 73014585868 block 0: fixing magic # to 0x58444233
bad directory block magic # 0x10a5bf9c for directory inode 42949675715 block 0: fixing magic # to 0x58444233
bad directory block magic # 0x9214d055 for directory inode 51539795872 block 0: fixing magic # to 0x58444233
bad directory block magic # 0x5b58346c for directory inode 57982059162 block 0: fixing magic # to 0x58444233
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
- 20:39:06: verify and correct link counts - 38 of 38 allocation groups done
Metadata corruption detected at 0x5609236cdcc8, xfs_dir3_block block 0xff00332e0/0x1000
libxfs_bwrite: write verifier failed on xfs_dir3_block bno 0xff00332e0/0x8
Metadata corruption detected at 0x5609236cdcc8, xfs_dir3_block block 0xf77fff228/0x1000
libxfs_bwrite: write verifier failed on xfs_dir3_block bno 0xf77fff228/0x8
Metadata corruption detected at 0x5609236cdcc8, xfs_dir3_block block 0x7f8007bf8/0x1000
libxfs_bwrite: write verifier failed on xfs_dir3_block bno 0x7f8007bf8/0x8
Metadata corruption detected at 0x5609236cdcc8, xfs_dir3_block block 0xe101f32f8/0x1000
libxfs_bwrite: write verifier failed on xfs_dir3_block bno 0xe101f32f8/0x8
Metadata corruption detected at 0x5609236cdcc8, xfs_dir3_block block 0x960000640/0x1000
libxfs_bwrite: write verifier failed on xfs_dir3_block bno 0x960000640/0x8
Metadata corruption detected at 0x5609236cdcc8, xfs_dir3_block block 0xca7fff978/0x1000
libxfs_bwrite: write verifier failed on xfs_dir3_block bno 0xca7fff978/0x8
Metadata corruption detected at 0x5609236cdcc8, xfs_dir3_block block 0xb4002d4f0/0x1000
libxfs_bwrite: write verifier failed on xfs_dir3_block bno 0xb4002d4f0/0x8
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Releasing dirty buffer to free list!
xfs_repair: Refusing to write a corrupt buffer to the data device!
xfs_repair: Lost a write to the data device!
fatal error -- File system metadata writeout failed, err=117. Re-run xfs_repair.
