On 21/10/2019 10:34, Ming Lei wrote:
On Mon, Oct 21, 2019 at 10:19:18AM +0100, John Garry wrote:
On 20/10/2019 11:14, Ming Lei wrote:
ght? If so, I need to find some simple sysfs entry to
tell me of this occurrence, to trigger the capture. Or add something. My
script is pretty dump.
BTW, I did notice that we the dump_stack in __blk_mq_run_hw_queue()
pretty soon before the problem happens - maybe a clue or maybe coincidence.
I finally got to capture that debugfs dump at the point the SCSI IOs
timeout, as attached. Let me know if any problem receiving it.
Here's a kernel log snippet at that point (I added some print for the
timeout):
609] psci: CPU6 killed.
[ 547.722217] CPU5: shutdown
[ 547.724926] psci: CPU5 killed.
[ 547.749951] irq_shutdown
[ 547.752701] IRQ 800: no longer affine to CPU4
[ 547.757265] CPU4: shutdown
[ 547.759971] psci: CPU4 killed.
[ 547.790348] CPU3: shutdown
[ 547.793052] psci: CPU3 killed.
[ 547.818330] CPU2: shutdown
[ 547.821033] psci: CPU2 killed.
[ 547.854285] CPU1: shutdown
[ 547.856989] psci: CPU1 killed.
[ 575.925307] scsi_timeout req=0xffff0023b0dd9c00 reserved=0
[ 575.930794] scsi_timeout req=0xffff0023b0df2700 reserved=0
From the debugfs log, 66 requests are dumped, and 63 of them has
been submitted to device, and the other 3 is in ->dispatch list
via requeue after timeout is handled.
Hi Ming,
You mentioned that:
" - I added some debug prints in blk_mq_hctx_drain_inflight_rqs() for when
inflights rqs !=0, and I don't see them for this timeout"
There might be two reasons:
1) You are still testing a multiple reply-queue device?
As before, I am testing by exposing mutliple queues to the SCSI midlayer. I
had to make this change locally, as on mainline we still only expose a
single queue and use the internal reply queue when enabling managed
interrupts.
As I
mentioned last times, it is hard to map reply-queue into blk-mq
hctx correctly.
Here's my branch, if you want to check:
https://github.com/hisilicon/kernel-dev/commits/private-topic-sas-5.4-mq-v4
It's a bit messy (sorry), but you can see that the reply-queue in the LLDD
is removed in commit 087b95af374.
I am now thinking of actually making this change to the LLDD in mainline to
avoid any doubt in future.
As I mentioned last time, you do share tags among all MQ queues on your hardware
given your hardware is actually SQ HBA, so commit 087b95af374 is definitely
wrong, isn't it?
Yes, we share tags among all queues, but we generate the tag - known as
IPTT - in the LLDD now, as we can no longer use the request tag (as it
is not unique per all queues):
https://github.com/hisilicon/kernel-dev/commit/087b95af374be6965583c1673032fb33bc8127e8#diff-f5d8fff19bc539a7387af5230d4e5771R188
As I said, the branch is messy and I did have to fix 087b95af374.
It can be very hard to partition the single tags among multiple hctx.
I really do think now that I'll make this change on mainline to avoid
doubt...
Thanks,
John
Thanks,
Ming
.
