On Thu, Sep 26, 2019 at 5:01 PM Bart Van Assche <bvanassche@xxxxxxx> wrote: > > On 9/26/19 2:55 AM, Roman Penyaev wrote: > > As I remember correctly I could not reuse the whole machinery with those > > restarts from block core because shared tags are shared only between > > hardware queues, i.e. different hardware queues share different tags sets. > > IBTRS has many hardware queues (independent RDMA connections) but only one > > tags set, which is equally shared between block devices. What I dreamed > > about is something like BLK_MQ_F_TAG_GLOBALLY_SHARED support in block > > layer. > > A patch series that adds support for sharing tag sets across hardware > queues is pending. See also "[PATCH V3 0/8] blk-mq & scsi: fix reply > queue selection and improve host wide tagset" > (https://lore.kernel.org/linux-block/20180227100750.32299-1-ming.lei@xxxxxxxxxx/). > Would that patch series allow to remove the queue management code from > ibnbd? Hi Bart, No, it seems this thingy is a bit different. According to my understanding patches 3 and 4 from this patchset do the following: 1# split equally the whole queue depth on number of hardware queues and 2# return tag number which is unique host-wide (more or less similar to unique_tag, right?). 2# is not needed for ibtrs, and 1# can be easy done by dividing queue_depth on number of hw queues on tag set allocation, e.g. something like the following: ... tags->nr_hw_queues = num_online_cpus(); tags->queue_depth = sess->queue_deph / tags->nr_hw_queues; blk_mq_alloc_tag_set(tags); And this trick won't work out for the performance. ibtrs client has a single resource: set of buffer chunks received from a server side. And these buffers should be dynamically distributed between IO producers according to the load. Having a hard split of the whole queue depth between hw queues we can forget about a dynamic load distribution, here is an example: - say server shares 1024 buffer chunks for a session (do not remember what is the actual number). - 1024 buffers are equally divided between hw queues, let's say 64 (number of cpus), so each queue is 16 requests depth. - only several CPUs produce IO, and instead of occupying the whole "bandwidth" of a session, i.e. 1024 buffer chunks, we limit ourselves to a small queue depth of an each hw queue. And performance drops significantly when number of IO producers is smaller than number of hw queues (CPUs), and it can be easily tested and proved. So for this particular ibtrs case tags should be globally shared, and seems (unfortunately) there is no any other similar requirements for other block devices. -- Roman
