On 6/3/19 1:38 PM, Hannes Reinecke wrote:
> On 6/3/19 11:31 AM, Ming Lei wrote:
>> On Mon, Jun 03, 2019 at 10:47:24AM +0200, Hannes Reinecke wrote:
>>> On 6/3/19 10:16 AM, Ming Lei wrote:
>>>> On Mon, Jun 03, 2019 at 09:46:39AM +0200, Hannes Reinecke wrote:
>>>>> On 6/3/19 9:37 AM, Ming Lei wrote:
>>>>>> On Mon, Jun 03, 2019 at 08:08:18AM +0200, Hannes Reinecke wrote:
>>>>>>> On 6/1/19 1:06 AM, Ming Lei wrote:
>>>>>>>> On Fri, May 31, 2019 at 12:26:56PM +0200, Hannes Reinecke wrote:
>>>>>>>>> On 5/31/19 10:46 AM, Ming Lei wrote:
>>>>>>> [ .. ]
>>>>>>>>> First we check for the 'slot_index_alloc()' callback to handle weird v2
>>>>>>>>> allocation rules, _then_ we look for a tag, and only if we do _not_ have
>>>>>>>>> a tag we're using the bitmap.
>>>>>>>>
>>>>>>>> OK, looks I miss the above change.
>>>>>>>>
>>>>>>>>> And the bitmap is already correctly sized, as otherwise we'd have a
>>>>>>>>> clash between internal and tagged I/O commands even now.
>>>>>>>>
>>>>>>>> But now the big problem is in the following two line code:
>>>>>>>>
>>>>>>>> + else if (blk_tag != (u32)-1)
>>>>>>>> + rc = blk_mq_unique_tag_to_tag(blk_tag);
>>>>>>>>
>>>>>>>> Request from different blk-mq hw queue has same tag returned from
>>>>>>>> blk_mq_unique_tag_to_tag().
>>>>>>>>
>>>>>>> Yes, but the sbitmap allocator will ensure that each command will get a
>>>>>>> unique tag.
>>>>>>
>>>>>> Each hw queue has independent sbitmap allocator, so commands with same
>>>>>> tag can come from different hw queue.
>>>>>>
>>>>> It does not for SCSI.
>>>>> See below.
>>>>>
>>>>>> So you meant this RFC patch depends on the host-wide tags patchset I
>>>>>> posted?
>>>>>>
>>>>>>>
>>>>>>>> Now the biggest question is that if V3 hw supports per-queue tags,
>>>>>>>> If yes, it should be real MQ hardware, otherwise I guess commands with
>>>>>>>> same tag at the same time may not work for host-wide tags.
>>>>>>>>
>>>>>>>
>>>>>>> Of course you can't have different commands with the same tag. But the
>>>>>>> sbitmap allocator prevents this from happening, as for host-wide tags
>>>>>>> the tagset is _shared_ between all devices, so the sbitmap allocator
>>>>>>> will only ever run on _one_ tagset for all commands.
>>>>>>
>>>>>> But blk-mq doesn't support host-wide tags yet, so how can this single
>>>>>> patch work?
>>>>>>
>>>>> Wrong. It does:
>>>>>
>>>>> struct request_queue *scsi_mq_alloc_queue(struct scsi_device *sdev)
>>>>> {
>>>>> sdev->request_queue = blk_mq_init_queue(&sdev->host->tag_set);
>>>>> if (IS_ERR(sdev->request_queue))
>>>>> return NULL;
>>>>>
>>>>> sdev->request_queue->queuedata = sdev;
>>>>> __scsi_init_queue(sdev->host, sdev->request_queue);
>>>>> blk_queue_flag_set(QUEUE_FLAG_SCSI_PASSTHROUGH, sdev->request_queue);
>>>>> return sdev->request_queue;
>>>>> }
>>>>>
>>>>>
>>>>> IE every scsi device is using the tagset from the host.
>>>>
>>>> Looks we are not in the same page, and you misunderstood two concepts:
>>>> scsi's host-wide tagset, and the new host-tags of BLK_MQ_F_HOST_TAGS.
>>>>
>>>> I admit that the new flag of BLK_MQ_F_HOST_TAGS is misleading.
>>>>
>>>> Now let me clarify it a bit:
>>>>
>>>> 1) the current SCSI hostwide tags means all LUNs share the host tagset,
>>>> but the tagset may include multiple hw queues, and each hw queue still
>>>> has independent tags, that is why blk-mq provides blk_mq_unique_tag().
>>>> In short, each LUN's hw queue has independent tags.
>>>>
>>> Which is where I fundamentally disagree.
>>> Each hw queue does _not_ have independent tags.
>>> Each hw queue will use tags from the same (host-wide) tagset; the tags
>>> themselves will be allocated for each queue on an ad-hoc base, ie there
>>> is no fixed mapping between tag values and hardware queues.
>>
>> Tagset is set of tags, and one tags is for serving one hw queue.
>>
>> Each hw queue has its own tags, please see __blk_mq_alloc_rq_map()
>> in which standalone sbitmap allocator and rq pool is allocated to
>> each hw queue represented by 'hctx_idx'.
>>
> Yes, but ...
>
>> And for each hw queue, the allocated tag value for request is in
>> the range of 0 ~ queue_depth - 1, that is why I say requests from
>> different hw queue may have same tag.
>>
>
> But this is not what I have been observing working with lpfc and qla2xxx.
> Both drivers have been converted to using scsi-mq with nr_hw_queues > 1
> some years ago, and do work just fine.
> And none of those drivers allow for re-using an in-flight tag on
> different hardware queues.
> If your reasoning is correct none of these drivers would work.
>
To illustrate what I mean I've attached a histogram from a fio run on a
smartpqi device (with 48 queues and 1024 tags in total).
As you can see, tags are evenly used by any queue, with the tag
allocator ensuring that in-flight tags will not be re-used on other hw
queues.
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@xxxxxxx +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Attachment:
tag-distribution.pdf
Description: Adobe PDF document
