Dear Linux-Raid and Linux-XFS experts:
I'm posting this on both the linux-raid and linux-xfs
mailing list as it's not clear at this point wether
this is a MD- od XFS-problem.
I have described my problem in a recent posting on
linux-raid and Wol's conclusion was:
In other words, one or more of the following three are true :-
1) The OP has been caught by some random act of God
2) There's a serious flaw in "mdadm --grow"
3) There's a serious flaw in xfs
Cheers,
Wol
There's very important data on our RAID10 device but I doubt
it's important enough for God to take a hand into our storage.
But let me first summarize what happened and why I believe that
this is an XFS-problem:
Machine running Linux 3.14.69 with no kernel-patches.
XFS filesystem was created with XFS userutils 3.1.11.
I did a fresh compile of xfsprogs-4.9.0 yesterday when
I realized that the 3.1.11 xfs_repair did not help.
mdadm is V3.3
/dev/md5 is a RAID10-device that was created in Feb 2013
with 10 2TB disks and an ext3 filesystem on it. Once in a
while I added two more 2TB disks. Reshaping was done
while the ext3 filesystem was mounted. Then the ext3
filesystem was unmounted resized and mounted again. That
worked until I resized the RAID10 from 16 to 20 disks and
realized that ext3 does not support filesystems >16TB.
I switched to XFS and created a 20TB filesystem. Here are
the details:
# xfs_info /dev/md5
meta-data=/dev/md5 isize=256 agcount=32,
agsize=152608128 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=4883457280, imaxpct=5
= sunit=128 swidth=1280 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Please notice: Ths XFS-filesystem has a size of
4883457280*4K = 19,533,829,120K
On saturday I tried to add two more 2TB disks to the RAID10
and the XFS filesystem was mounted (and in medium use) at that
time. Commands were:
# mdadm /dev/md5 --add /dev/sdo
# mdadm --grow /dev/md5 --raid-devices=21
# mdadm -D /dev/md5
/dev/md5:
Version : 1.2
Creation Time : Sun Feb 10 16:58:10 2013
Raid Level : raid10
Array Size : 19533829120 (18628.91 GiB 20002.64 GB)
Used Dev Size : 1953382912 (1862.89 GiB 2000.26 GB)
Raid Devices : 21
Total Devices : 21
Persistence : Superblock is persistent
Update Time : Sat Jan 6 15:08:37 2018
State : clean, reshaping
Active Devices : 21
Working Devices : 21
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Reshape Status : 1% complete
Delta Devices : 1, (20->21)
Name : backup:5 (local to host backup)
UUID : 9030ff07:6a292a3c:26589a26:8c92a488
Events : 86002
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 65 48 1 active sync /dev/sdt
2 8 64 2 active sync /dev/sde
3 65 96 3 active sync /dev/sdw
4 8 112 4 active sync /dev/sdh
5 65 144 5 active sync /dev/sdz
6 8 160 6 active sync /dev/sdk
7 65 192 7 active sync /dev/sdac
8 8 208 8 active sync /dev/sdn
9 65 240 9 active sync /dev/sdaf
10 65 0 10 active sync /dev/sdq
11 66 32 11 active sync /dev/sdai
12 8 32 12 active sync /dev/sdc
13 65 64 13 active sync /dev/sdu
14 8 80 14 active sync /dev/sdf
15 65 112 15 active sync /dev/sdx
16 8 128 16 active sync /dev/sdi
17 65 160 17 active sync /dev/sdaa
18 8 176 18 active sync /dev/sdl
19 65 208 19 active sync /dev/sdad
20 8 224 20 active sync /dev/sdo
Please notice: Ths RAID10-device has a size of 19,533,829,120K
that's exactly the same size as the contained XFS-filesystem.
Immediately after the RAID10 reshape operation started the
XFS-filesystem reported I/O-errors and was severly damaged.
I waited for the reshape operation to finish and tried to repair
the filesystem with xfs_repair (version 3.1.11) but xfs_repair
crashed, so I tried 4.9.0-version aif xfs_reapair with no luck
either.
/dev/md5 ist now mounted ro,norecovery with an overlay filesystem
on top of it (thanks very much to Andreas for that idea) and I have
setup a new server today. Rsyncing the data to the new server will
take a while and I'm sure I will stumble on lots of corrupted files.
I proceeded from XFS to ZFS (skipped YFS) so lengthy reshape
operations won't happen in the future anymore.
Here are the relevant log messages:
Jan 6 14:45:00 backup kernel: md: reshape of RAID array md5
Jan 6 14:45:00 backup kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jan 6 14:45:00 backup kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
Jan 6 14:45:00 backup kernel: md: using 128k window, over a total of 19533829120k.
Jan 6 14:45:00 backup kernel: XFS (md5): metadata I/O error: block 0x12c08f360 ("xfs_trans_read_buf_map") error 5 numblks 16
Jan 6 14:45:00 backup kernel: XFS (md5): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
Jan 6 14:45:00 backup kernel: XFS (md5): metadata I/O error: block 0x12c08f360 ("xfs_trans_read_buf_map") error 5 numblks 16
Jan 6 14:45:00 backup kernel: XFS (md5): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
... hundreds of the above XFS-messages deleted
Jan 6 14:45:00 backup kernel: XFS (md5): Log I/O Error Detected. Shutting down filesystem
Jan 6 14:45:00 backup kernel: XFS (md5): Please umount the filesystem and rectify the problem(s)
Please notice: no error message about hardware-problems.
All 21 disks are fine and the next messages from the
md-driver was:
Jan 7 02:28:02 backup kernel: md: md5: reshape done.
Jan 7 02:28:03 backup kernel: md5: detected capacity change from 20002641018880 to 21002772807680
I'm wondering about one thing: the first xfs message is about a
meatadata I/O error on block 0x12c08f360. Since the xfs filesystem
has a blocksize of 4K this block is located at position 20135005568K
which is beyond the end of the RAID10 device. No wonder that the
xfs driver receives an I/O error. And also no wonder that the
filesystem is severely corrupted right now.
Question 1: How did the xfs driver knew on Jan 6 that the RAID10
device was about to be increased from 20TB to 21TB on Jan 7?
Question 2: Why did the xfs driver started to use the additional
space that was not yet there without me executing xfs_growfs.
This looks like a severe XFS-problem to me.
But my hope is that all the data taht was within the filesystem
before Jan 6 14:45 is not involved in the corruption. If xfs
started to use space beyond the end of the underlying raid
device this should have affected only data that was created,
modified or deleted after Jan 6 14:45.
If that was true we could clearly distinct between data
that we must dump and data that we can keep. The machine is
our backup system (as you may have guessed from its name)
and I would like to keep old backup-files.
I remember that mkfs.xfs is clever enough to adopt the
filesystem paramters to the underlying hardware of the
block device that the xfs filesystem is created on. Hence
from the xfs drivers point of view the underlying block
device is not just a sequence of data blocks, but the xfs
driver knows something about the layout of the underlying
hardware.
If that was true - how does the xfs driver reacts if that
information about the layout of the underlying hardware
changes while the xfs-filesystem is mounted?
Seems to be an interesting problem
Kind regards
Peter Koch
--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
