In Qian's kernel .config, async scsi scan is disabled so in failure
case SCSI scan type is synchronous.
Below is the stack trace when scsi_scan_host() hangs:
[<0>] __wait_rcu_gp+0x134/0x170
[<0>] synchronize_rcu.part.80+0x53/0x60
[<0>] blk_free_flush_queue+0x12/0x30
[<0>] blk_mq_hw_sysfs_release+0x21/0x70
this is per blk_mq_hw_ctx
[<0>] kobject_release+0x46/0x150
[<0>] blk_mq_release+0xb4/0xf0
[<0>] blk_release_queue+0xc4/0x130
[<0>] kobject_release+0x46/0x150
[<0>] scsi_device_dev_release_usercontext+0x194/0x3f0
[<0>] execute_in_process_context+0x22/0xa0
[<0>] device_release+0x2e/0x80
[<0>] kobject_release+0x46/0x150
[<0>] scsi_alloc_sdev+0x2e7/0x310
[<0>] scsi_probe_and_add_lun+0x410/0xbd0
[<0>] __scsi_scan_target+0xf2/0x530
[<0>] scsi_scan_channel.part.7+0x51/0x70
[<0>] scsi_scan_host_selected+0xd4/0x140
[<0>] scsi_scan_host+0x198/0x1c0
This issue hits when lock related debugging is enabled in kernel config.
kernel .config parameters(may be subset of this list) are required to
hit the issue:
CONFIG_PREEMPT_COUNT=y *
CONFIG_UNINLINE_SPIN_UNLOCK=y *
CONFIG_LOCK_STAT=y
CONFIG_DEBUG_RT_MUTEXES=y *
CONFIG_DEBUG_SPINLOCK=y *
CONFIG_DEBUG_MUTEXES=y *
CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y *
CONFIG_DEBUG_RWSEMS=y *
CONFIG_DEBUG_LOCK_ALLOC=y *
CONFIG_LOCKDEP=y *
CONFIG_DEBUG_LOCKDEP=y
CONFIG_TRACE_IRQFLAGS=y *
CONFIG_TRACE_IRQFLAGS_NMI=y
CONFIG_DEBUG_KOBJECT=y
CONFIG_PROVE_RCU=y *
CONFIG_PREEMPTIRQ_TRACEPOINTS=y *
(* means that I enabled)
When scsi_scan_host() hangs, there are no outstanding IOs with
megaraid_sas driver-firmware stack as SCSI "host_busy" counter and
megaraid_sas driver's internal counter are "0".
Key takeaways:
1. Issue is observed when lock related debugging is enabled so issue
is seen in debug environment.
2. Issue seems to be related to generic shared "host_tagset" code
whenever some kind of kernel debugging is enabled. We do not see an
immediate reason to hide this issue through disabling the
"host_tagset" feature.
John,
Issue may hit on ARM platform too using Qian's .config file with other
adapters (e.g. hisi_sas) as well. So I feel disabling “host_tagset” in
megaraid_sas driver will not help. It requires debugging from the
“Entire Shared host tag feature” perspective as scsi_scan_host()
waittime aggravates when "host_tagset" is enabled. Also, I am doing
parallel debugging and if I find anything useful, I will share.
So isn't this then really related to how many HW queues we expose there
is just scaling up the time? For megaraid sas, it's 1->128 for my arm64
platform when host_tagset_enable=1.
As a hack, I tried this (while keeping host_tagset_enable=1):
@@ -6162,11 +6168,15 @@ static int megasas_init_fw(struct
megasas_instance *instance)
else
instance->low_latency_index_start = 1;
- num_msix_req = num_online_cpus() +
instance->low_latency_index_start;
+ num_msix_req = 6 + instance->low_latency_index_start;
(6 is an arbitrary small number)
And boot time is nearly same as with host_tagset_enable=0.
For hisi_sas, max HW queue number ever is 16. In addition, we don't scan
each channel/id/lun for hisi_sas, as it has a scan handler.
Qian,
I need full dmesg logs from your setup with
megaraid_sas.host_tagset_enable=1 and
megaraid_sas.host_tagset_enable=0. Please wait for a long time. I just
want to make sure that whatever you observe is the same as mine.
Thanks,
John
