> On Aug 27, 2024, at 15:24, Muchun Song <muchun.song@xxxxxxxxx> wrote:
>
>
>
>> On Aug 26, 2024, at 17:20, Ming Lei <ming.lei@xxxxxxxxxx> wrote:
>>
>> On Mon, Aug 26, 2024 at 03:33:18PM +0800, Muchun Song wrote:
>>>
>>>
>>>> On Aug 26, 2024, at 15:06, Muchun Song <songmuchun@xxxxxxxxxxxxx> wrote:
>>>>
>>>> On Fri, Aug 23, 2024 at 7:28 PM Ming Lei <ming.lei@xxxxxxxxxx> wrote:
>>>>>
>>>>> On Sun, Aug 11, 2024 at 06:19:21 PM +0800, Muchun Song wrote:
>>>>>> Supposing the following scenario.
>>>>>>
>>>>>> CPU0 CPU1
>>>>>>
>>>>>> blk_mq_request_issue_directly() blk_mq_unquiesce_queue()
>>>>>> if (blk_queue_quiesced()) blk_queue_flag_clear(QUEUE_FLAG_QUIESCED) 3) store
>>>>>> blk_mq_insert_request() blk_mq_run_hw_queues()
>>>>>> /* blk_mq_run_hw_queue()
>>>>>> * Add request to dispatch list or set bitmap of if (!blk_mq_hctx_has_pending()) 4) load
>>>>>> * software queue. 1) store return
>>>>>> */
>>>>>> blk_mq_run_hw_queue()
>>>>>> if (blk_queue_quiesced()) 2) load
>>>>>> return
>>>>>> blk_mq_sched_dispatch_requests()
>>>>>>
>>>>>> The full memory barrier should be inserted between 1) and 2), as well as
>>>>>> between 3) and 4) to make sure that either CPU0 sees QUEUE_FLAG_QUIESCED is
>>>>>> cleared or CPU1 sees dispatch list or setting of bitmap of software queue.
>>>>>> Otherwise, either CPU will not re-run the hardware queue causing starvation.
>>>>>
>>>>> Memory barrier shouldn't serve as bug fix for two slow code paths.
>>>>>
>>>>> One simple fix is to add helper of blk_queue_quiesced_lock(), and
>>>>> call the following check on CPU0:
>>>>>
>>>>> if (blk_queue_quiesced_lock())
>>>>> blk_mq_run_hw_queue();
>>>>
>>>> This only fixes blk_mq_request_issue_directly(), I think anywhere that
>>>> matching this
>>>> pattern (inserting a request to dispatch list and then running the
>>>> hardware queue)
>>>> should be fixed. And I think there are many places which match this
>>>> pattern (E.g.
>>>> blk_mq_submit_bio()). The above graph should be adjusted to the following.
>>>>
>>>> CPU0 CPU1
>>>>
>>>> blk_mq_insert_request() 1) store blk_mq_unquiesce_queue()
>>>> blk_mq_run_hw_queue()
>>>> blk_queue_flag_clear(QUEUE_FLAG_QUIESCED) 3) store
>>>> if (blk_queue_quiesced()) 2) load blk_mq_run_hw_queues()
>>>> return blk_mq_run_hw_queue()
>>>> blk_mq_sched_dispatch_requests() if
>>>> (!blk_mq_hctx_has_pending()) 4) load
>>>> return
>>>
>>> Sorry. There is something wrong with my email client. Resend the graph.
>>>
>>> CPU0 CPU1
>>>
>>> blk_mq_insert_request() 1) store blk_mq_unquiesce_queue()
>>> blk_mq_run_hw_queue() blk_queue_flag_clear(QUEUE_FLAG_QUIESCED) 3) store
>>> if (blk_queue_quiesced()) 2) load blk_mq_run_hw_queues()
>>> return blk_mq_run_hw_queue()
>>> blk_mq_sched_dispatch_requests() if (!blk_mq_hctx_has_pending()) 4) load
>>> return
>>
>> OK.
>>
>> The issue shouldn't exist if blk_queue_quiesced() return false in
>> blk_mq_run_hw_queue(), so it is still one race in two slow paths?
>>
>> I guess the barrier-less approach should work too, such as:
>>
>
> If we prefer barrier-less approach, I think the following solution
> will work as well, I'll use it in v2. Thanks.
>
>>
>> diff --git a/block/blk-mq.c b/block/blk-mq.c
>> index e3c3c0c21b55..632261982a77 100644
>> --- a/block/blk-mq.c
>> +++ b/block/blk-mq.c
>> @@ -2202,6 +2202,12 @@ void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs)
>> }
>> EXPORT_SYMBOL(blk_mq_delay_run_hw_queue);
>>
>> +static inline bool blk_mq_hw_queue_need_run(struct blk_mq_hw_ctx *hctx)
>> +{
>> + return !blk_queue_quiesced(hctx->queue) &&
>> + blk_mq_hctx_has_pending(hctx);
>> +}
>> +
>> /**
>> * blk_mq_run_hw_queue - Start to run a hardware queue.
>> * @hctx: Pointer to the hardware queue to run.
>> @@ -2231,11 +2237,19 @@ void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async)
>> * quiesced.
>> */
>> __blk_mq_run_dispatch_ops(hctx->queue, false,
>> - need_run = !blk_queue_quiesced(hctx->queue) &&
>> - blk_mq_hctx_has_pending(hctx));
>> + need_run = blk_mq_hw_queue_need_run(hctx));
>>
>> - if (!need_run)
>> - return;
>> + if (!need_run) {
>> + unsigned long flags;
>> +
>> + /* sync with unquiesce */
>> + spin_lock_irqsave(&hctx->queue->queue_lock, flags);
After some time thought, I think here we need a big comment to explain
why we need to sync. Because there are other caller of blk_queue_quiesced()
which do not need to hold ->queue_lock to sync. Then, I am thinking
is ->queue_lock really easier to be maintained than mb? For developers,
we still need to care about this, right? I don't see any obvious benefit.
And the mb approach seems more efficient than spinlock. Something like:
if (!need_run) {
/* Add a comment here to explain what's going on here. */
smp_mb();
need_run = blk_mq_hw_queue_need_run(hctx);
if (!need_run)
return;
}
I am not objecting to your approach, I want to know if you insist on
barrier-less approach here. If yes, I'm fine with this approach. I can
use it in v2.
Muhcun,
Thanks.
>> + need_run = blk_mq_hw_queue_need_run(hctx);
>
> One question here: should we use __blk_mq_run_dispatch_ops()? I saw a comment above.
> It seems it is safe to call blk_mq_hw_queue_need_run under [s]rcu lock.
>
> Thanks.
>
>> + spin_unlock_irqrestore(&hctx->queue->queue_lock, flags);
>> +
>> + if (!need_run)
>> + return;
>> + }
>>
>> if (async || !cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)) {
>> blk_mq_delay_run_hw_queue(hctx, 0);
>>
>>
>> thanks,
>> Ming
>
>
