Re: [PATCH RFC] hisi_sas_v3: multiqueue support

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 6/3/19 1:38 PM, Hannes Reinecke wrote:
> On 6/3/19 11:31 AM, Ming Lei wrote:
>> On Mon, Jun 03, 2019 at 10:47:24AM +0200, Hannes Reinecke wrote:
>>> On 6/3/19 10:16 AM, Ming Lei wrote:
>>>> On Mon, Jun 03, 2019 at 09:46:39AM +0200, Hannes Reinecke wrote:
>>>>> On 6/3/19 9:37 AM, Ming Lei wrote:
>>>>>> On Mon, Jun 03, 2019 at 08:08:18AM +0200, Hannes Reinecke wrote:
>>>>>>> On 6/1/19 1:06 AM, Ming Lei wrote:
>>>>>>>> On Fri, May 31, 2019 at 12:26:56PM +0200, Hannes Reinecke wrote:
>>>>>>>>> On 5/31/19 10:46 AM, Ming Lei wrote:
>>>>>>> [ .. ]
>>>>>>>>> First we check for the 'slot_index_alloc()' callback to handle weird v2
>>>>>>>>> allocation rules, _then_ we look for a tag, and only if we do _not_ have
>>>>>>>>> a tag we're using the bitmap.
>>>>>>>>
>>>>>>>> OK, looks I miss the above change.
>>>>>>>>
>>>>>>>>> And the bitmap is already correctly sized, as otherwise we'd have a
>>>>>>>>> clash between internal and tagged I/O commands even now.
>>>>>>>>
>>>>>>>> But now the big problem is in the following two line code:
>>>>>>>>
>>>>>>>> +       else if (blk_tag != (u32)-1)
>>>>>>>> +               rc = blk_mq_unique_tag_to_tag(blk_tag);
>>>>>>>>
>>>>>>>> Request from different blk-mq hw queue has same tag returned from
>>>>>>>> blk_mq_unique_tag_to_tag().
>>>>>>>>
>>>>>>> Yes, but the sbitmap allocator will ensure that each command will get a
>>>>>>> unique tag.
>>>>>>
>>>>>> Each hw queue has independent sbitmap allocator, so commands with same
>>>>>> tag can come from different hw queue.
>>>>>>
>>>>> It does not for SCSI.
>>>>> See below.
>>>>>
>>>>>> So you meant this RFC patch depends on the host-wide tags patchset I
>>>>>> posted?
>>>>>>
>>>>>>>
>>>>>>>> Now the biggest question is that if V3 hw supports per-queue tags,
>>>>>>>> If yes, it should be real MQ hardware, otherwise I guess commands with
>>>>>>>> same tag at the same time may not work for host-wide tags.
>>>>>>>>
>>>>>>>
>>>>>>> Of course you can't have different commands with the same tag. But the
>>>>>>> sbitmap allocator prevents this from happening, as for host-wide tags
>>>>>>> the tagset is _shared_ between all devices, so the sbitmap allocator
>>>>>>> will only ever run on _one_ tagset for all commands.
>>>>>>
>>>>>> But blk-mq doesn't support host-wide tags yet, so how can this single
>>>>>> patch work?
>>>>>>
>>>>> Wrong. It does:
>>>>>
>>>>> struct request_queue *scsi_mq_alloc_queue(struct scsi_device *sdev)
>>>>> {
>>>>> 	sdev->request_queue = blk_mq_init_queue(&sdev->host->tag_set);
>>>>> 	if (IS_ERR(sdev->request_queue))
>>>>> 		return NULL;
>>>>>
>>>>> 	sdev->request_queue->queuedata = sdev;
>>>>> 	__scsi_init_queue(sdev->host, sdev->request_queue);
>>>>> 	blk_queue_flag_set(QUEUE_FLAG_SCSI_PASSTHROUGH, sdev->request_queue);
>>>>> 	return sdev->request_queue;
>>>>> }
>>>>>
>>>>>
>>>>> IE every scsi device is using the tagset from the host.
>>>>
>>>> Looks we are not in the same page, and you misunderstood two concepts:
>>>> scsi's host-wide tagset, and the new host-tags of BLK_MQ_F_HOST_TAGS.
>>>>
>>>> I admit that the new flag of BLK_MQ_F_HOST_TAGS is misleading.
>>>>
>>>> Now let me clarify it a bit:
>>>>
>>>> 1) the current SCSI hostwide tags means all LUNs share the host tagset,
>>>> but the tagset may include multiple hw queues, and each hw queue still
>>>> has independent tags, that is why blk-mq provides blk_mq_unique_tag().
>>>> In short, each LUN's hw queue has independent tags.
>>>>
>>> Which is where I fundamentally disagree.
>>> Each hw queue does _not_ have independent tags.
>>> Each hw queue will use tags from the same (host-wide) tagset; the tags
>>> themselves will be allocated for each queue on an ad-hoc base, ie there
>>> is no fixed mapping between tag values and hardware queues.
>>
>> Tagset is set of tags, and one tags is for serving one hw queue.
>>
>> Each hw queue has its own tags, please see __blk_mq_alloc_rq_map()
>> in which standalone sbitmap allocator and rq pool is allocated to
>> each hw queue represented by 'hctx_idx'.
>>
> Yes, but ...
> 
>> And for each hw queue, the allocated tag value for request is in
>> the range of 0 ~ queue_depth - 1, that is why I say requests from
>> different hw queue may have same tag.
>>
> 
> But this is not what I have been observing working with lpfc and qla2xxx.
> Both drivers have been converted to using scsi-mq with nr_hw_queues > 1
> some years ago, and do work just fine.
> And none of those drivers allow for re-using an in-flight tag on
> different hardware queues.
> If your reasoning is correct none of these drivers would work.
> 
To illustrate what I mean I've attached a histogram from a fio run on a
smartpqi device (with 48 queues and 1024 tags in total).
As you can see, tags are evenly used by any queue, with the tag
allocator ensuring that in-flight tags will not be re-used on other hw
queues.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@xxxxxxx			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)

Attachment: tag-distribution.pdf
Description: Adobe PDF document


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [SCSI Target Devel]     [Linux SCSI Target Infrastructure]     [Kernel Newbies]     [IDE]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux ATA RAID]     [Linux IIO]     [Samba]     [Device Mapper]

  Powered by Linux