I have used the slicing scheme. A couple of advantages are that the smaller grows will finish in a reasonable time (vs a single grow/replace that takes a week or more). And the drives often have small ranges of sectors that have read errors causing only a single partition to get thrown out and have to be re-added and that smaller chunk rebuild much faster. I have also had SATA cabling/interface issues that caused a few temporary failures to happen and those errors do not generally result in all the partitions getting kicked out, leaving a much smaller re-add. And another advantage is that you aren't limited to the exact same sizes of devices. I use LVM on a 4 section raid built very similar to his, except mine is a linear lvm so no raid0 seek issues. I think we need to see a grep -E 'md5|sdk' /var/log/messages to see where sdk went. I have never see devices get removed straight away, mine go to failed and then I have to remove them and re-add them when one does have an issue. On Wed, Nov 23, 2022 at 4:41 PM Roman Mamedov <rm@xxxxxxxxxxx> wrote: > > On Wed, 23 Nov 2022 22:07:36 +0000 > David T-G <davidtg-robot@xxxxxxxxxxxxxxx> wrote: > > > diskfarm:~ # mdadm -D /dev/md50 > > /dev/md50: > > Version : 1.2 > > Creation Time : Thu Nov 4 00:56:36 2021 > > Raid Level : raid0 > > Array Size : 19526301696 (18.19 TiB 19.99 TB) > > Raid Devices : 6 > > Total Devices : 6 > > Persistence : Superblock is persistent > > > > Update Time : Thu Nov 4 00:56:36 2021 > > State : clean > > Active Devices : 6 > > Working Devices : 6 > > Failed Devices : 0 > > Spare Devices : 0 > > > > Layout : -unknown- > > Chunk Size : 512K > > > > Consistency Policy : none > > > > Name : diskfarm:10T (local to host diskfarm) > > UUID : cccbe073:d92c6ecd:77ba5c46:5db6b3f0 > > Events : 0 > > > > Number Major Minor RaidDevice State > > 0 9 51 0 active sync /dev/md/51 > > 1 9 52 1 active sync /dev/md/52 > > 2 9 53 2 active sync /dev/md/53 > > 3 9 54 3 active sync /dev/md/54 > > 4 9 55 4 active sync /dev/md/55 > > 5 9 56 5 active sync /dev/md/56 > > It feels you haven't thought this through entirely. Sequential writes to this > RAID0 array will alternate across all member arrays, and seeing how those are > not of independent disks, but instead are "vertical" across partitions on the > same disks, it will result in a crazy seek load, as first 512K is written to > the array of the *51 partitions, second 512K go to *52, then to *53, > effectively requiring a full stroke of each drive's head across the entire > surface for each and every 3 *megabytes* written. > > mdraid in the "linear" mode, or LVM with one large LV across all PVs (which > are the individual RAID5 arrays), or multi-device Btrfs using "single" profile > for data, all of those would avoid the described effect. > > But I should clarify, the entire idea of splitting drives like this seems > questionable to begin with, since drives more often fail entirely, not in part, > so you will not save any time on rebuilds; and the "bitmap" already protects > you against full rebuilds due to any hiccups such as a power cut; or even if a > drive failed in part, in your current setup, or even in the proposed ones I > mentioned above, losing even one RAID5 of all these, would result in a > complete loss of data anyway. Not to mention what you have seems like an insane > amount of complexity. > > To summarize, maybe it's better to blow away the entire thing and restart from > the drawing board, while it's not too late? :) > > > diskfarm:~ # mdadm -D /dev/md5[13456] | egrep '^/dev|active|removed' > > /dev/md51: > > 0 259 9 0 active sync /dev/sdb51 > > 1 259 2 1 active sync /dev/sdc51 > > 3 259 16 2 active sync /dev/sdd51 > > - 0 0 3 removed > > /dev/md53: > > 0 259 11 0 active sync /dev/sdb53 > > 1 259 4 1 active sync /dev/sdc53 > > 3 259 18 2 active sync /dev/sdd53 > > - 0 0 3 removed > > /dev/md54: > > 0 259 12 0 active sync /dev/sdb54 > > 1 259 5 1 active sync /dev/sdc54 > > 3 259 19 2 active sync /dev/sdd54 > > - 0 0 3 removed > > /dev/md55: > > 0 259 13 0 active sync /dev/sdb55 > > 1 259 6 1 active sync /dev/sdc55 > > 3 259 20 2 active sync /dev/sdd55 > > - 0 0 3 removed > > /dev/md56: > > 0 259 14 0 active sync /dev/sdb56 > > 1 259 7 1 active sync /dev/sdc56 > > 3 259 21 2 active sync /dev/sdd56 > > - 0 0 3 removed > > > > that are obviously the sdk (new disk) slice. If md52 were also broken, > > I'd figure that the disk was somehow unplugged, but I don't think I can > > plug in one sixth of a disk and leave the rest unhooked :-) So ... In > > addition to wondering how I got here, how do I remove the "removed" ones > > and then re-add them to build and grow and finalize this? > > If you want to fix it still, without dmesg it's hard to say how this could > have happened, but what does > > mdadm --re-add /dev/md51 /dev/sdk51 > > say? > > -- > With respect, > Roman
