On Tue, Mar 3, 2020 at 5:23 PM John Garry <john.garry@xxxxxxxxxx> wrote: > > >> And for these low-latency queues, I figure that the issue is not just > >> polling vs interrupt, but indeed how fast the HW queue can consume SQEs. > >> A HW queue may only be able to consume at a limited rate - that's why we > >> segregate. > > Yes, there is no polling in any of HW queues. High IOPs queues have > > interrupt coalescing enabled whereas > > low latency queues does not have interrupt coalescing. megaraid_sas > > driver would choose which set of queues > > among these two has to be used depending on workload. For latency > > oriented workload, driver would use low > > latency queues and for IOPs profile, driver would use High IOPs queues. > >> > >> As an aside, that is actually an issue for blk-mq. For 1 to many HW > >> queue-to-CPU mapping, limiting many CPUs a single queue can limit IOPs > >> since HW queues can only consume at a limited rate. > > We were able to achieve performance target for MegaRAID latest gen > > controller with this model of few set > > of HW queues mapped to local numa CPUs and low latency queues has one > > to one mapping to CPUs. > > This is default behavior of queues segregation in megaraid_sas driver > > to satisfy our IOPs and latency requirements altogether. > > However we have given module parameter- "perf_mode" to tune queues > > behavior. i.e turning on/off interrupt > > coalescing on all HW queues where this one to many queues to CPU > > mapping would not happen. > > Hi Sumit, Hi John, > > OK, I have a rough idea of the concept. And again I'd say megaraid sas > may not be a good candidate to expose > 1 HW queues, as we hide HW > queues and don't maintain the symmetry with blk-mq layer. Sorry, my last response was not very clear. I was referring to reply queues as HW queues not submission queues. I agree with you, since megaraid_sas HW has single submission queue so >1 HW queue would not help to improve performance. Testing done by us on shared tagset patches worked by you/Hannes was to ensure no performance drop from single HW submission queue based driver. > > Indeed, I do not even expect a performance increase in exposing > 1 HW > queues since the driver already uses the reply map + managed interrupts. > > The main reason for that change in some drivers - apart from losing the > duplicated ugliness of the reply map - is to leverage the blk-mq feature > to drain a hctx for CPU hotplug [0] - is this something which megaraid > sas is vulnerable to and would benefit from? "megaraid_sas" driver would be benefited with draining of IO completions directed to hotplugged(offlined) CPU. With current driver IO completion would hang, if CPU on which IO is to be completed goes offline. Thanks, Sumit > > Thanks, > John > > [0] > https://lore.kernel.org/linux-block/20200115114409.28895-1-ming.lei@xxxxxxxxxx/
