Matt, I have put as many as 32 partitions on a drive (based upon great advice from this list) and done RAID6 over them, but I was concerned about our sustainability long term. As a researcher, I can do these cool science experiments, but I still have to hand designs to sustainment folks. I was also running into an issue of doing a mdraid RAID0 on top of the RAID6's so I could toss one xfs file system on top each of the numa node's drives and the last RAID0 stripe of all of the RAID6's couldn't generate the queue depth needed. We even recompiled the kernel to change the mdraid nr_request max from 128 to 1023. I will have to try the LVM experiment. I'm an LVM neophyte, so it might take me the rest of today/tomorrow to get new results as I tend to let mdraid do all of its volume builds without forcing, so that will take a bit of time also. Once might be able to argue that configuration isn't too much of a "Frankenstein's monster" for me to hand it off. Thanks, Jim -----Original Message----- From: Matt Wallis <mattw@xxxxxxxxxxxx> Sent: Wednesday, July 28, 2021 6:32 AM To: Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx> Cc: linux-raid@xxxxxxxxxxxxxxx Subject: [Non-DoD Source] Re: Can't get RAID5/RAID6 NVMe randomread IOPS - AMD ROME what am I missing????? Hi Jim, > On 28 Jul 2021, at 06:32, Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx> wrote: > > Sorry, this will be a long email with everything I find to be relevant, but I can get over 110GB/s of 4kB random reads from individual NVMe SSDs, but I'm at a loss why mdraid can only do a very small fraction of it. I'm at my "organizational world record" for sustained IOPS, but I need protected IOPS to do something useful. This is everything I do to a server to make the I/O crank.....My role is that of a lab researcher/resident expert/consultant. I'm just stumped why I can't do better. If there is a fine manual that somebody can point me to, I'm happy to read it… I am probably going to get corrected on some if not all of this, but from what I understand, and from my own little experiments on a similar Intel based system… 1. NVMe is stupid fast, you need a good chunk of CPU performance to max it out. 2. Most block IO in the kernel is limited in terms of threading, it may even be essentially single threaded. (This is where I will get corrected) 3. AFAICT, this includes mdraid, there’s a single thread per RAID device handling all the RAID calculations. (mdX_raid6) What I did to get IOPs up in a system with 24 NVMe, split up into 12 per NUMA domain. 1. Create 8 partitions on each drive (this may be overkill, I just started here for some reason) 2. Create 8 RAID6 arrays with 1 partition per drive. 3. Use LVM to create a single striped logical volume over all 8 RAID volumes. RAID 0+6 as it were. You now have an LVM thread that is basically doing nothing more than chunking the data as it comes in, then, sending the chunks to 8 separate RAID devices, each with their own threads, buffers, queues etc, all of which can be spread over more cores. I saw a significant (for me, significant is >20%) increase in IOPs doing this. You still have RAID6 protection, but you might want to write a couple of scripts to help you manage the arrays, because a single failed drive now needs to be removed from 8 RAID volumes. There’s not a lot of capacity lost doing this, pretty sure I lost less than 100MB to the partitions and the RAID overhead. You would never consider this on spinning disk of course, way to slow and you’re just going to make it slower, NVMe as you noticed has the IOPs to spare, so I’m pretty sure it’s just that we’re not able to get the data to it fast enough. Matt
