Hello all, On Mon, 2020-11-16 at 16:51 +0000, Finlayson, James M CIV (USA) wrote: > On Wed, Oct 28, 2020 at 6:39 PM Vitaly Mayatskih < > v.mayatskih@xxxxxxxxx> wrote > > > > On Thu, Oct 22, 2020 at 2:56 AM Finlayson, James M CIV (USA) < > > james.m.finlayson4.civ@xxxxxxxx> wrote: > > > > > > All, > > > I'm working on creating raid5 or raid6 arrays of 800K IOP nvme > > > drives. Each of \ > > > the drives performs well with a queue depth of 128 and I set to > > > 1023 if allowed. \ > > > In order for me to try to max out the queue depth on each RAID > > > member, so I'd like \ > > > to set the sysfs nr_requests on the md device to something greater > > > than 128, like \ > > > #raid members * 128. Even though > > > /sys/block/md127/queue/nr_requests is mode 644, \ > > > when I try to change nr_requests in any way as root, I get write > > > error: invalid \ > > > argument. When I'm hitting the md device with random reads, my > > > nvme drives are \ > > > 100% utilized, but only doing 160K IOPS because they have no queue > > > depth. > > > Am I doing something silly? > > > > It only works for blk-mq block devices. MD is not blk-mq. Would it be possible to implement something similar to the use_blk_mq of dm_mod on md_mod? > > > > You can exchange simplicity for performance: instead of creating one > > RAID-5/6 array you can partition drives in N equal sized partitions, > > create N RAID-5/6 arrays using one partition from every disk, then > > stripe them into top-level RAID-0. So that would be RAID-5+0 (or > > 6+0). > > > > It is awful, but simulates multiqueue and performs better in > > parallel > > loads. Especially for writes (on RAID-5/6). > > > > > > -- > > wbr, Vitaly > > Vitaly, > Thank you for the tip. My raid5 performance (after creating 32 > partitions per SSD) and running 64 9+1 (2 in reality) stripes is up > to 11.4M 4K random read IOPS, out of 17M that the box is capable, > which I'm happy with, because I can't NUMA the raid stripes as I would > the individual SSDs themselves. However, when I perform the RAID0 > striping to make the "RAID50 from hell", my performance drops to 7.1M > 4K random read IOPS. Any suggestions? The last RAID50, again won't > let me generate the queue depth. > > Thanks in advance, > Jim > > >
