On Thu, Oct 22, 2020 at 2:56 AM Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx> wrote: > > All, > I'm working on creating raid5 or raid6 arrays of 800K IOP nvme drives. Each of the drives performs well with a queue depth of 128 and I set to 1023 if allowed. In order for me to try to max out the queue depth on each RAID member, so I'd like to set the sysfs nr_requests on the md device to something greater than 128, like #raid members * 128. Even though /sys/block/md127/queue/nr_requests is mode 644, when I try to change nr_requests in any way as root, I get write error: invalid argument. When I'm hitting the md device with random reads, my nvme drives are 100% utilized, but only doing 160K IOPS because they have no queue depth. > > Am I doing something silly? It only works for blk-mq block devices. MD is not blk-mq. You can exchange simplicity for performance: instead of creating one RAID-5/6 array you can partition drives in N equal sized partitions, create N RAID-5/6 arrays using one partition from every disk, then stripe them into top-level RAID-0. So that would be RAID-5+0 (or 6+0). It is awful, but simulates multiqueue and performs better in parallel loads. Especially for writes (on RAID-5/6). -- wbr, Vitaly
