Hi, Actually, the RAID5/RAID6 mdraid implementations can't support the IOPS or the queue depths required for a basic single mdraid raid5/raid6 LUN. The partitions are just to create more mdraid stripes out of partitions to allow for more threads? To do the RAID parity work and to be able to issue more I/Os to the entirety of the NVMe SSDs using mdraid. Ultimately, I need one volume per NUMA domain comprised of RAIDed NVME SSDs. We're just exploring creative workarounds to the NVMe mdraid IOPS issues to get the most IOPS out of a collection of SSDs. I still have to put an xfs file system per volume for something useful to occur. Thanks, Jim -----Original Message----- From: Wols Lists <antlists@xxxxxxxxxxxxxxx> Sent: Thursday, July 29, 2021 12:35 PM To: Matt Wallis <mattw@xxxxxxxxxxxx>; Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx> Cc: linux-raid@xxxxxxxxxxxxxxx Subject: Re: [Non-DoD Source] Can't get RAID5/RAID6 NVMe randomread IOPS - AMD ROME what am I missing????? On 29/07/21 01:54, Matt Wallis wrote: > Hi Jim, > > Totally get the Frankenstein’s monster aspect, I try not to build those where I can, but at the moment I don’t think there’s much that can be done about it. > Not sure if LVM is better than MDRAID 0, it just gives you more control over the volumes that can be created, instead of having it all in one big chunk. If you just need one big chunk, then MDRAID 0 is probably fine. > sticking raid 0 on top of raid 6 sounds an extremely weird thing to do. What I guess you might be wanting to do instead is write a partition table to the raid-6? That's perfectly normal if, imho, a bit unusual? And LVM would be MUCH better than raid-0, I'm sure, because it addresses this very issue by design, rather than by accident. > I think if you can create a couple of scripts that allows the admin to fail a drive out of all the arrays that it’s in at once, then it's not that much worse than managing an MDRAID is normally. Is that wise? KISS. > > Matt. > >> On 28 Jul 2021, at 20:43, Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx> wrote: >> >> Matt, >> I have put as many as 32 partitions on a drive (based upon great advice from this list) and done RAID6 over them, but I was concerned about our sustainability long term. As a researcher, I can do these cool science experiments, but I still have to hand designs to sustainment folks. I was also running into an issue of doing a mdraid RAID0 on top of the RAID6's so I could toss one xfs file system on top each of the numa node's drives and the last RAID0 stripe of all of the RAID6's couldn't generate the queue depth needed. We even recompiled the kernel to change the mdraid nr_request max from 128 to 1023. >> >> I will have to try the LVM experiment. I'm an LVM neophyte, so it might take me the rest of today/tomorrow to get new results as I tend to let mdraid do all of its volume builds without forcing, so that will take a bit of time also. Once might be able to argue that configuration isn't too much of a "Frankenstein's monster" for me to hand it off. >> Do. If it solves what you want, then it's worth it. I'm moving my stuff over to LVM. To throw something else into the mix, you've gone for raid 6, which enables you to lose two drives, or corrupt one drive. Do you need the two-drive redundancy? The calculations are a lot more expensive than raid-5 if you're worried over write speed. I don't know the impact of it but I'm playing with dm-integrity which provides some protection against corruption. >> Thanks, >> Jim >> Cheers, Wol
